Graph Scattering beyond Wavelet Shackles
Anonymous Author(s)
AfﬁliationAddress email
Abstract
This work develops a ﬂexible and mathematically sound framework for the designand analysis of graph scattering networks with variable branching ratios and genericfunctional calculus ﬁlters.
Spectrally-agnostic stability guarantees for node- andgraph-level perturbations are derived; the vertex-set non-preserving case is treatedby utilizing recently developed mathematical-physics based tools.
Energy propagation through the network layers is investigated and related to truncation stability.
New methods of graph-level feature aggregation are introduced and stability ofthe resulting composite scattering architectures is established.
Finally, scatteringtransforms are extended to edge- and higher order tensorial input.
Theoreticalresults are complemented by numerical investigations: Suitably chosen scatteringnetworks conforming to the developed theory perform better than traditional graphwavelet based scattering approaches in social network graph classiﬁcation tasksand signiﬁcantly outperform other graph-based learning approaches to regressionof quantum-chemical energies on QM 7 .
Introduction
Euclidean wavelet scattering networks [23, 4] are deep convolutional architectures where outputfeatures are generated in each layer.
Employed ﬁlters are designed rather than learned and derivefrom a ﬁxed (tight) wavelet frame, resulting in a tree structured network with constant branching ratio.
Such networks provide state of the art methods in settings with limited data availability and serveas a mathematically tractable model of standard convolutional neural networks (CNNs).
Rigorousinvestigations — establishing remarkable invariance- and stability properties of wavelet scatteringnetworks — were initially carried out in [23].
The extensive mathematical analysis [40] generalizedthe term ’scattering network’ to include tree structured networks with varying branching rations andframes of convolutional ﬁlters, thus signiﬁcantly narrowing the conceptual gap to general CNNs.
With increasing interest in data on graph-structured domains, well performing networks generalizing
Euclidean CNNs to this geometric setting emerged [18, 5, 9].
If efﬁciently implemented, such graphconvolutional networks (GCNs) replace Euclidean convolutional ﬁlters by functional calculus ﬁlters;i.e.
scalar functions applied to a suitably chosen graph-shift-oprator capturing the geometry of theunderlying graph [18, 14, 9].
Almost immediately, proposals aimed at extending the success story of
Euclidean scattering networks to the graph convolutional setting began appearing: In [50], the authorsutilize dyadic graph wavelets (see e.g. [14]) based on the non-normalized graph Laplacian resultingin a norm preserving graph wavelet scattering transform.
In [10], diffusion wavelets (see e.g. [8]) areused to construct a graph scattering transform enjoying spectrum-dependent stability guarantees tograph level perturbations.
For scattering transforms with N layers and K distinct functional calculusﬁlters, the work [11] derives node-level stability bounds of O p K N { 2 q and conducts correspondingnumerical experiments choosing diffusion wavelets, monic cubic wavelets [14] and tight Hannwavelets [36] as ﬁlters.
In [12] the authors, following [8], construct so called geometric wavelets andestablish the expressivity of a scattering transform based on such a frame through extensive numerical experiments.
A theoretical analysis of this and a closely related wavelet based scattering transform isthe main focus of [29].
Additionally, graph-wavelet based scattering transforms have been extendedto the spatio-temporal domain [28], utilized to overcome the problem of oversmoothing in GCNs [26]and pruned to deal with their exponential (in network depth) increase in needed computing resources[15].
Common among all these contributions is the focus on graph wavelets, which are genericallyunderstood to derive in a scale-sampling procedure from a common wavelet generating kernelfunction g : R Ñ R satisfying various properties [14].
Established stability or expressivity properties— especially to structural perturbations — are then generally linked to the speciﬁc choice of thewavelet kernel g and utilized graph shift operator [10, 29].
This severely limits the diversity ofavailable ﬁlter banks in the design of scattering networks and draws into question their validity asmodels for more general GCNs whose ﬁlters generically do not derive from a wavelet kernel.
A primary focus of this work is to provide alleviation in this situation: After reviewing the graph signalprocessing setting in Section 2, we introduce a general framework for the construction of (generalized)graph scattering transforms beyond the wavelet setting in Section 3.
Section 4 establishes spectrumagnostic stability guarantees on the node signal level and for the ﬁrst time also for graph-levelperturbations.
To handle the vertex-set non-preserving case, a new ’distance measure’ for operatorscapturing the geometry of varying graphs is utilized.
After providing conditions for energy decay(with the layers) and relating it to truncation stability, we consider graph level feature aggregationand higher order inputs in Sections 5 and 6 respectively.
In Section 7 we then provide numericalresults indicating that general functional calculus ﬁlter based scattering is at least as expressive asstandard wavelet based scattering in graph classiﬁcation tasks and outperforms leading graph neuralnetwork approaches to regression of quantum chemical energies on QM 7 .
Graph Signal Processing
Taking a signal processing approach, we consider signals on graphs as opposed to graph embeddings:
Node-Signals: Given a graph p G, E q , we are primarily interested in node-signals, which arefunctions from the node-set G to the complex numbers, modelled as elements of C | G | .
We equip thisspace with an inner product according to x f, g y
“ ř | G | i “ 1 f i g i µ i (with all vertex weights µ i ě 1 ) anddenote the resulting inner product space by (cid:96) 2 p G q .
We forego considering arbitrary inner products on
C | G | solely in the interest of increased readability.
Functional Calculus Filters: Our fundamental objects in investigating node-signals will be functional calculus ﬁlters based on a normal operator ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p G q .
Prominent examples includethe adjacency matrix W , the degree matrix D , normalized p
´ D ´ 12 WD ´ 12 q or un-normalized( L : “ D ´ W ) graph Laplacians Writing normalized eigenvalue-eigenvector pairs of ∆ as p
λ
i , φ i q | G | i “ 1 ,the ﬁlter obtained from applying g : C Ñ C is given by g p ∆ q f “ ř | G | i “ 1 g p λ
i qx φ
i , f y (cid:96) 2 p V q φ i .
Theoperator we utilize in our numerical investigations of Section 6, is given by L : “ L { λ max p L q .
Wedivide by the largest eigenvalue to ensure that the spectrum σ p L q is contained in the interval r 0 , 1 s ,which aids in the choice of functions from which ﬁlters are derived.
Generalized Frames: We are most interested in ﬁlters that arise from a collection of functionsadequately covering the spectrum of the operator to which they are applied.
To this end we call acollection t g i p¨qu i P I of functions a generalized frame if it satisﬁes the generalized frame condition
A ď ř
i P I
| g i p c q| 2 ď B for any c in C for constants A ; B ą 0 .
As proved in Appendix B, thiscondition is sufﬁcient to guarantee that the associated operators form a frame:
Theorem 2.1.
Let ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p
G q be normal.
If the family t g i p¨qu i P I of bounded functionssatisﬁes A ď
ř
i
P
I | g
i p c q| 2 ď B for all c in the spectrum σ p ∆ q , we have ( @ f P (cid:96) 2 p G q )
[Equation]
Notably, the functions t g i u i P
I need not be continuous:
In fact, in our numerical implementations, wewill – among other mappings – utilize the function δ 0 p¨q , deﬁned by δ 0 p 0 q “ 1 and δ
p c q “ 0 forc ‰ 0 as well as a modiﬁed cosine, deﬁned by cos p 0 q “ 0 and cos p c q “ cos p c q for c ‰ 0 .
The Generalized Graph Scattering Transform
A generalized graph scattering transform is a non-linear map Φ based on a tree structured multilayergraph convolutional network with constant branching factor in each layer.
For an input signalf P (cid:96) 2 p G q , outputs are generated in each layer of such a scattering network, and then concatenated toform a feature vector in a feature space F .
The network is built up from three ingredients:
Connecting Operators: To allow intermediate signal representations in the ’hidden’ networklayers to be further processed with functional calculus ﬁlters based on varying operators, which mightnot all be normal for the same choice of node-weights, we allow these intermediate representationsto live in varying graph signal spaces.
In fact, we do not even assume that these signal spaces arebased on a common vertex set.
This is done to allow for modelling of recently proposed networkswhere input- and ’processing’ graphs are decoupled (see e.g. [1, 37]), as well as architecturesincorporating graph pooling [21].
Instead, we associate one signal space (cid:96) 2 p G n q to each layern .
Connecting operators are then (not necessarily linear) operators P n : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n qconnecting the signal spaces of subsequent layers.
We assume them to be Lipschitz continuous( } P p f q ´ P p g q} (cid:96) 2 p G n ´ 1 q ď R ` } f ´ g } (cid:96) 2 p G n q q and triviality preserving ( P p 0 q “ 0 ).
For our originalnode-signal space we also write (cid:96) 2 p G q ” (cid:96) 2 p G 0 q .
Non-Linearities:
To each layer, we also associate a (possibly) non-linear function ρ n : C Ñ Cacting poinwise on signals in (cid:96) 2 p G n q .
Similar to connecting operators, we assume ρ n preserves zeroand is Lipschitz-continuous with Lipschitz constant denoted by L ` n .
This deﬁnition allows for theabsolute value non-linearity, but also ReLu or – trivially – the identity function.
Operator Frames: Beyond these ingredients, the central building block of our scatteringarchitecture is comprised of a family of functional calculus ﬁlters in each layer.
That is, weassume that in each layer, the node signal space (cid:96) 2 p G n q carries a normal operator ∆ n and anassociated collection of functions comprised of an output generating function χ n p¨q as wellas a ﬁlter bank t g γ n p¨qu γ n P Γ n indexed by an index set Γ n .
As the network layer n varies (andin contrast to wavelet-scattering networks)
we allow the index set Γ n as well as the collectiont χ n p¨qu Ť t g γ n p¨qu γ n P Γ n of functions to vary.
We only demand that in each layer the functions in theﬁlter bank together with the output generating function constitute a generalized frame with frameconstants A n , B n ě 0 .
We refer to the collection of functions Ω N : “ p ρ n , t χ n p¨qu Ť t g γ n p¨qu γ n P Γ n q Nn “ 1 as a module sequence and call D N : “ p P n , ∆ n q Nn “ 1 our operator collection .
The generalized scatteringtransform is then constructed iteratively:
To our initial signal f P (cid:96) 2 p G q we ﬁrst apply the connecting operator P 1 , yielding a signal representation in (cid:96) 2 p G 1 q .
Subsequently, we apply the pointwise non-linearity ρ 1 .
Then we apply our graph ﬁlters t χ 1 p ∆ 1 qu Ť
[Equation]
1 to ρ 1 p P 1 p
f
qq yielding the output V 1 p f q : “ χ 1 p ∆ 1 q ρ 1 p P 1 p f qq as well as the intermediate hidden representations t U 1 r γ 1 sp f q : “ g γ 1 p ∆ 1 q ρ 1 p P 1 p f qqu γ 1 P Γ 1 obtained in the ﬁrst layer.
Here we have introduced the one-step scattering propagator U n r γ n s : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n q mapping f ÞÑ g γ n p ∆ n q ρ n p P n p f qq as well as the output generating operator
V n : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n q mapping f to χ n p ∆ n q ρ n p P n p f qq .
Upon deﬁning the set Γ N ´ 1 : “ Γ N ´ 1 ˆ ... ˆ Γ 1 of paths of length p N ´ 1 q terminating in layer N ´ 1 (with Γtaken to be the one-element set) and iterating the above procedure, we see that the outputs generated in the N th -layer are indexed by paths Γ N ´terminating in the previous layer.
Outputs generated in the N th layer are thus given by t V N ˝
U r γ N ´ 1 s˝ ...
˝ U r
[Equation]
1 sp f qu p
[Equation]
N
[Equation] [Equation]
1 qP
[Equation]
N
[Equation]
Concatenating the features obtained in the various layers of a network with depth N , our full featurevectors thus live in the feature space
[Equation]
The associated canonical norm is denoted } ¨ } F N .
For convenience, a brief review of direct sums ofspaces, their associated norms and a discussion of corresponding direct sums of maps is provided in
Appendix A. We denote the hence constructed generalized scattering transform of length N , basedon a module sequence Ω N and operator collection D N by Φ N .
In our numerical experiments in Section 7, we consider two particular instantiations of the above general architecture.
In both cases the utilized shift-operator is L : “ L { λ max p L q , node weights satisfy µ i “ 1 , the branching ratio in each layer is chosen as 4 and the depth is set to N “ 4 as well.
The connecting operators are set to the identity and non-linearities are set to the modulus ( |¨| ).
The two architectures differ in the utilized ﬁlters, which are repeated in each layer and depicted in Fig.
Postponing a discussion of other parameterchoices, we note here that the ﬁlters t sin p π { 2 ¨q , cos p π { 2 ¨qu provide a high and a low pass ﬁlter on the spectrum σ p L q Ď r 0 , 1 s , while t sin p π ¨q , cos p π ¨qu provides a spectral reﬁnement of the former two ﬁlters.
The inner two elements of the ﬁlter bank in Architecture II thus separate an input signal into high- and low-lying spectral
components.
The outer two act similarly at a higher spectral scale.
Additionally Architecture I –utilizing cos and δ 0 as introduced Section 2 – prevents the lowest lying spectral information frompropagating.
Instead it is extracted via δ
p¨q in each layer.
Note that Id arises from applying theconstant- 1 function to L .
Normalizations are chosen to generate frames with upper bounds B ž 1 .
Stability Guarantees
In order to produce meaningful signal representations, a small change in input signal should producea small change in the output of our generalized scattering transforms.
This property is captured in theresult below, which is proved in Appendix C.
Theorem 4.1.
With the notation of Section 3, we have for all f, h P (cid:96) 2 p G q :
[Equation]
In the case where upper frame bounds B n and Lipschitz constants L ` n and R ` n are all smaller than orequal one, this statement reduces to the much nicer inequality:
[Equation]
Below, we always assume R ` n , L ` n ď 1 as this easily achievable through rescaling.
We will keep B nvariable to demonstrate how ﬁlter size inﬂuences stability results.
As for our experimentally testedarchitectures (cf.
Fig. 2), we note for Architecture I that B n “ 1 { 2 for all n , so that (2) applies.
For
Architecture II we have B n “ 3 , which yields a stability constant of ?
1 ` 2 ¨ 3 ` 2 ¨
3 2 ` 2 ¨ 3 3 “ 9 .
Similar to other constants derived in this section, this bound is however not necessarily tight.
Operators capturing graph geometries might only be known approximately in real world tasks; e.g. ifedge weights are only known to a certain level of precision.
Hence it is important that our scatteringrepresentation be insensitive to small perturbations in the underlying normal operators in each layer,which is captured by our next result, proved in Appendix D. Smallness here is measured in Frobeniusnorm } ¨ } F , which for convenience is brieﬂy reviewed in Appendix A).
Theorem 4.2.
Let Φ N and r
Φ N be two scattering transforms based on the same module sequenceΩ N and operator sequences D N , r D N with the same connecting operators ( P n “ r P n ) in eachlayer.
Assume R ` n , L ` n ď 1 and B n ď B for some B and n ď N .
Assume that the respectivenormal operators satisfy } ∆ n ´ r ∆ n } F ď δ for some δ ą 0 .
Further assume that the functions t g γ n u γ n P Γ n and χ n in each layer are Lipschitz continuous with associated Lipschitz constantssatisfying L 2 χ n ` ř
[Equation]
ď D 2 for all n ď N and some D ą 0 .
Then we have
[Equation]
for all f P (cid:96) 2 p G q .
If B ď 1 { 2 , the stability constant improves to
[Equation]
The condition B ď 12 is e.g. satisﬁed by our Architecture I, but –strictly speaking– we may notapply Theorem 4.2, since not all utilized ﬁlters are Lipschitz continuous.
Remark D.3 in Appendix Dhowever shows, that the above stability result remains applicable for this architecture as long as wedemand that ∆ and r ∆ are (potentially rescaled) graph Laplacians.
For Architecture II we note that
D “ π ? 10 { 2 and thus the stability constant is given by
[Equation]
We are also interested in perturbations that change the vertex set of the graphs in our architecture.
This is important for example in the context of social networks, when passing from nodes representingindividuals to nodes representing (close knit) groups of individuals.
To investigate this setting, weutilize tools originally developed within the mathematical physics community [30]:
Deﬁnition 4.3.
Let H and r H be two ﬁnite dimensional Hilbert spaces.
Let ∆ and r ∆ be normaloperators on these spaces.
Let J : H Ñ r H and r J : r H Ñ H be linear maps — called identiﬁcationoperators .
We call the two spaces δ -quasi-unitarily-equivalent (with δ ě 0 ) if for any f P H andu P r H we have
[Equation]
ď 2 } f } H , }p J ´ r J ˚ q f } r H ď δ } f } H , } f ´ r JJf } H ď δ b } f } 2 H ` x f, | ∆ | f y H , } u ´ J r Ju } r H ď δ b } u } 2 r H ` x u, | r ∆ | u y r H .
If, for some w P C the resolvent R : “ p ∆ ´ ω q ´ 1 satisﬁes }p r RJ ´ JR q f } r H ď δ } f } H for all f P H ,we say that ∆ and r ∆ are ω - δ -close with identiﬁcation operator J .
Absolute value | ∆ | and adjoint r J ˚ of operators are brieﬂy reviewed in
Appendix A. While the above deﬁnition might seem fairly abstract at ﬁrst, it is in fact a natural setting to investigate structural perturbations as Figure 3 exempliﬁes.
In our current setting, the Hilbert spaces in Deﬁnition 4.3 are node-signal spaces H “ (cid:96) 2 p G q , r H “ (cid:96) 2 p r G q of different graphs.
The notion of ω - δ -closeness is then useful, as it allows to compare ﬁlters deﬁned on different graphs but obtained from applying the same function to the respective graph-operators:
Lemma 4.4.
In the setting of Deﬁnition 4.3 let ∆ and r ∆ be ω - δclose and satisfy } ∆ } op , } r ∆ } op ď K for some K ą 0 .
If g : C Ñ C is holomorphic on the disk B K ` 1 p 0 q of radius p K ` 1 q , there is a constant C g ě
so that
[Equation]
with C g depending on g , ω and K .
An explicit characterization of C g together with a proof of this result is presented in Appendix F. Lemma 4.4 is our main tool in establishing our next result, proved in Appendix G, which captures stability under vertex-set non-preserving perturbations:
Figure
Details in Appendix E
Theorem 4.5.
Let Φ N , r Φ N be scattering transforms based on a common module sequence Ω N anddiffering operator sequences D N , r D N .
Assume R ` n , L ` n ď 1 and B n ď B for some B and n ě 0 .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q , r J n : (cid:96) 2 p r G n q Ñ (cid:96) 2 p G n q( 0 ď n ď N ) so that the respective signal spaces are δ -unitarily equivalent, the respective normaloperators ∆ n , r ∆ n are ω - δ -close as well as bounded (in norm) by K ą 0 and the connectingoperators satisfy } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q “ 0 .
For the common module sequence Ω N assumethat the non-linearities satisfy } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q “ 0 and that the constants C χ n and t C g γn u γ n P Γ N associated through Lemma 4.4 to the functions of the generalized frames in each layersatisfy C 2 χ n ` ř γ n P Γ N C 2 g γn ď D 2 for some D ą 0 .
Denote the operator that the family t J n u n ofidentiﬁcation operators induce on F N through concatenation by J N : F N Ñ Ă F N .
Then, with
K N “ a
[Equation]
if B
[Equation]
1 { 2 and K N “ a
[Equation] [Equation]
The stability result persists with slightly altered stability constants, if identiﬁcation operators onlyalmost commute with non-linearities and/or connecting operators, as Appendix G further elucidates.
Theorem 4.5 is not applicable to Architecture I, where ﬁlters are not all holomorphic, but is directlyapplicable to Architecture II.
Stability constants can be calculated in terms of D and B as before.
Beyond these results, stability under truncation of the scattering transform is equally desirable:
Giventhe energy W N : “ ř p γ N ,...,γ 1 qP Γ N } U r γ N s ˝ ...
˝
U r γ 1 sp f q}
[Equation]
2 p G N q stored in the network at layer N ,it is not hard to see that after extending Φ N p f q by zero to match dimensions with Φ N ` 1 p f
q
we have}
[Equation]
ď `
[Equation]
L ` N
[Equation]
˘
[Equation]
B N ` 1 ¨ W N (see Appendix H for more details).
Abound for W N is then given as follows:
Theorem 4.6.
Let Φ 8 be a generalized graph scattering transform based on a an operator sequence 193 D 8 “ p P n , ∆ n q 8 n “ 1 and a module sequence Ω 8 with each ρ n p¨q ě 0 .
Assume in each layer n ě 1that there is an eigenvector ψ n of ∆ n with solely positive entries; denote the smallest entry by m n : “min i P G n ψ n r
i s and the eigenvalue corresponding to ψ
n by λ n .
Quantify the ’spectral-gap’ opened upat this eigenvalue through neglecting the output-generating function by η n : “ ř γ n P Γ n | g γ n p λ n q| 2and assume B n m n ě η n .
We then have (with C ` N : “ ś Ni “ 1 max
[Equation] [Equation]
The product in (3) decays if C ` N Ñ C ` converges and ř Nn “ 1 p m n ´ η n { B n q Ñ 8 diverges as
N Ñ 8 .
The positivity-assumptions on the eigenvectors ψ n can e.g. always be ensured if they arechosen to lie in the lowest lying eigenspace of a graph Laplacian or normalized graph Laplacian(irrespective of the connectedness of the underlying graphs).
As an example, we note that if we extendour Architecture I to inﬁnite depth (recall from Section 3 that we are using the same ﬁlters, operators,etc.
in each layer) we have upon choosing λ n “ 0 and ψ n to be the constant normalized vector thatη n “ 0 , C N “ 1 and m n “ 1 { a | G | , for a graph with | G | vertices.
On a graph with 16 vertices, wethen e.g. have W N ď p 3 { 4 q N } f } 2 (cid:96) 2 p G q and thus } Φ N p f q ´ Φ N `
[Equation]
As detailed in Appendix H, Theorem 4.6 also implies that under the given assumptions the scatteringtransform has trivial ’kernel’ for N Ñ 8 , mapping only 0 to 0 .
Graph-Level Feature Aggregation
To solve tasks such as graph classiﬁcation or regression over multiple graphs, we need to representgraphs of varying sizes in a common feature space.
Given a scattering transform Φ N , we thusneed to ﬁnd a stability preserving map from the feature space F N to some Euclidean space that isindependent of any vertex set cardinalities.
Since F N is a large direct sum of smaller spaces (cf.
(1)),we simply construct such maps on each summand independently and then concatenate them.
General non-linear feature aggregation: Our main tool in passing to graph-level features is anon-linear map N Gp : (cid:96) 2 p G q Ñ R p given as
[Equation]
(4) with µ G : “ ř i P G µ i and } f } (cid:96) q p G q : “ p ř i P G | f i | q µ i q 1 { q .
Our inspiration to use this map stems fromthe standard case where all µ i “ 1 : For p ě | G | , the vector | f | “ pp| f 1 | , ..., | f G |q J can then berecovered from N Gp p f q up to permutation of indices [24].
Hence, employing N Gp (with p ě | G | ) toaggregate node-information into graph-level information, we lose the minimal necessary informationabout node permutation (clearly N Gp p f q “ N Gp p Π f q for any permutation matrix Π ) and beyond thatonly information about the complex phase (
Given a scattering transform Φ N mapping from (cid:96) 2 p G q to the feature space F N “ ‘ Nn “` (cid:96) 2 p G n q ˘ | Γ n ´ 1 | , we obtain a corresponding map Ψ N mapping from (cid:96) 2 p G q to
R N “ ‘ Nn “ 1 p R p n q | Γ n ´ 1 | by concatenating the feature map Φ N with the operator that the family of non-linear maps t N p n G n u Nn “ 1 induces on F N by concatenation.
Similarly we obtain the map r Ψ N : (cid:96) 2 p r G q Ñ R N by concatenating the map r Φ N : (cid:96) 2 p r G q Ñ ‘ Nn “´ (cid:96) 2 p r G n q ¯ | Γ n ´ 1 | with the operator induced by the family t N p n r G n u Nn “ 1 .
The feature space R N is completely determined by path-sets Γ N and used maximal p -norm indices p n .
It no longer depends on cardinalities of vertex sets of any graphs, allowing to compare (signals on) varying graphs with each other.
Most of the results of the previous sections then readily transfer to the graph-level-feature setting (c.f. Appendix I.1).
Low-pass feature aggregation:
The spectrum-free aggregation scheme of the previous paragraphis especially adapted to settings where there are no high-level spectral properties remaining constantunder graph perturbations.
However, many commonly utilized operators, such as normalized andun-normalized graph Laplacians, have a somewhat ’stable’ spectral theory: Eigenvalues are alwaysreal, non-negative, the lowest-lying eigenvalue equals zero and simple (if the graph is connected).
Inthis section we shall thus assume that each mentioned normal operator ∆ n ( r ∆ n ) has these spectralproperties.
We denote the lowest lying normalized eigenvector (which is generically determined upto a complex phase) by ψ ∆ n and denote by M |x¨ , ¨y| G n : (cid:96) 2 p G n q Ñ C the map given by M |x¨ , ¨y| G n p f q “|x ψ ∆ n , f y (cid:96) 2 p G n q | .
The absolute value around the inner product is introduced to absorb the phaseambiguity in the choice of ψ ∆ n .
Given a scattering transform Φ N mapping from (cid:96) 2 p G q to the featurespace F N , we obtain a corresponding map Ψ |x¨ , ¨y| N mapping from (cid:96) 2 p G q to C N “ ‘ Nn “ 1 C | Γ n ´ 1 | byconcatenating the feature map Φ N with the operator that the family of maps t M |x¨ , ¨y| G n u Nn “ 1 induces on 235 F
N by concatenation.
As detailed in Appendix I.2, this map inherits stability properties in completeanalogy to the discussion of Section 4.
Higher Order Scattering
Node signals capture information about nodes in isolation.
However, one might be interested inbinary, ternary or even higher order relations between nodes such as distances or angles in graphsrepresenting molecules.
In this section we focus on binary relations – i.e. edge level input – as this isthe instantiation we also test in our regression experiment in Section 7.
Appendix J.2 provides moredetails and extends these considerations beyond the binary setting.
We equip the space of edge inputswith an inner product according to x f, g y
“ ř | G | i,j “ 1 f ij g ij µ ij and denote the resulting inner-productspace by (cid:96) 2 p E q with E “ G ˆ G the set of edges.
Setting e.g. node-weights µ i and edge weights µ ik toone, the adjacency matrix W as well as normalized or un-normalized graph Laplacians constitute selfadjoint operators on (cid:96) 2 p E q , where they act by matrix multiplication.
Replacing the G n of Section 3by E n , we can then follow the recipe laid out there in constructing 2 nd -order scattering transforms; allthat we need are a module sequence Ω N and an operator sequence D 2 N : “ p P 2 n , ∆ 2 n q Nn “ 1 , where now
P 2 n : (cid:96) 2 p E n ´ 1 q Ñ (cid:96) 2 p E n q and ∆ 2 n : (cid:96) 2 p E n q Ñ (cid:96) 2 p E n q .
We denote the resulting feature map by Φ 2 Nand write F 2 N for the corresponding feature space.
The map N Gp introduced in (4) can also be adaptedto aggregate higher-order features into graph level features: With } f } q : “ p ř ij P G | f ij | q µ ij q 1 { q andµ E : “ ř | G | ij “ 1 µ ij , we deﬁne N Ep p
f q “ p} f } (cid:96) 1 p E q {? µ E , } f } (cid:96) 2 p E q , } f } (cid:96) 3 p E q , ..., } f } (cid:96)
p p E q q J {?
p .
Given a feature map Φ 2 N with feature space F 2 N “ ‘ Nn “` (cid:96) 2 p E n q ˘ | Γ n ´ 1 | , we obtain a correspondingmap Ψ 2 N mapping from (cid:96) 2 p E q to R N “ ‘ Nn “ 1 p R p n q | Γ n ´ 1 | by concatenating Φ EN with the map thatthe family of non-linear maps t N E n p n u Nn “ 1 induces on F N by concatenation.
The stability results ofthe preceding sections then readily translate to Φ 2 N and Ψ 2 N (c.f. Appendix J).
Experimental Results
We showcase that even upon selecting the fairly simple Architectures I and II introduced in Section3 (c.f. also Fig. 2), our generalized graph scattering networks are able to outperform both waveletbased scattering transforms and leading graph-networks under different circumstances.
To aid visualclarity when comparing results, we colour-code the best-performing method in green, the second-bestperforming in yellow and the third-best performing method in orange respectively.
Social Network Graph Classiﬁcation: To facilitate contact between our generalized graph scattering networks, and the wider literature, we combine a network conforming to our general theorynamely Architecture I in Fig.
2 (as discussed in Section 3 with depth N “ 4 , identity as connecting operators and | ¨ | -non-linearities) with the low pass aggregation scheme of Section 5 and a
Euclidean support vector machine with RBF-kernel (GGSN+EK).
The choice N “ 4 was madeto keep computation-time palatable, while aggregation scheme and non-linearities were chosento facilitate comparison with standard wavelet -scattering approaches.
For this hybrid architecture(GGSN+EK), classiﬁcation accuracies under the standard choice of 10 -fold cross validation on ﬁvecommon social network graph datasets are compared with performances of popular graph kernelapproaches, leading deep learning methods as well as geometric wavelet scattering (GS-SVM)
More details are provided in Appendix K.
As evident from Table 1, our network consistently achieveshigher accuracies than the geometric wavelet scattering transform of [12], with the performance gapbecoming signiﬁcant on the more complex REDDIT datasets, reaching a relative mean performanceincrease of more than 10% on REDDIT- 12 K. This indicates the liberating power of transcendingthe graph wavelet setting.
While on comparatively smaller and somewhat simpler datasets there isa performance gap between our static architecture and fully trainable networks, this gap closes onmore complex datasets: While P-Poinc e.g. outperforms our method on IMDB datasets, the rolesare reversed on REDDIT datasets.
On REDDIT-B our approach trails only GIN; with difference inaccuracies insigniﬁcant.
On REDDIT- 5 K our method comes in third, with the gap to the second bestmethod (GIN) being statistically insigniﬁcant.
On REDDIT- 12 K we generate state of the art results
[Table] [Table]
Regression of Quantum Chemical Energies: In order to showcase the prowess of both our higherorder scattering scheme and our spectrum-agnostic aggregation method of Section 5, we combinethese building blocks into a hybrid architecture which we then apply in combination with kernelmethods ( 2 GGST + EK) to the task of atomization energy regression on QM 7 .
This is a comparativelysmall dataset of 7165 molecular graphs, taken from the 970 million strong molecular database GDB13 [2].
Each graph in QM 7 represents an organic molecule, with nodes corresponding to individualatoms.
Beyond the node-level information of atomic charge, there is also edge level informationcharacterising interaction strengths between individual nodes/atoms available.
This is encoded intoso called Coulomb matrices (see e.g. [32] or Appendix K) of molecular graphs, which for us serve adual purpose: On the one hand we consider a Coulomb matrix as an edge-level input signal on a givengraph.
On the other hand, we also treat it as an adjacency matrix from which we build up a graph
Laplacian L .
Our normal operator is then chosen as L “ L { λ max p L q again.
Connecting operators areset to the identity, while non-linearities are ﬁxed to ρ n ě 1 p¨q “ |¨| .
Filters are chosen as p sin p π { 2 ¨ L q ,cos p π { 2 ¨ L q , sin p π ¨ L q , cos p π ¨ L qq acting through matrix multiplication,.
Output generatingfunctions are set to the identity and depth is N “ 4 , so that we essentially recover Architecture II of
Fig.
2; now applied to edge-level input.
Graph level features are aggregated via the map N E 5 of Section 6.
We chose p “ 5 (and not p ąą 5 ) for N G 2 p to avoid overﬁtting.
Generated feature vectors are combined with node node level scattering features obtained from applying Architecture II of Fig.
2 to the input signal of atomic charge into composite feature vectors; plotted in Figure 5.
As is visually evident, even when reduced to the low-dimensional subspace of their ﬁrst three principal components, the generated scattering features are able to aptly resolve the atomization energy of the molecules.
This aptitude is also reﬂected in Table 2, comparing our approach with leading graph-based learning methods trained with ten-fold cross validation on node 300 and (depending on the model) edge level information.
Our method is the best performing.
We signiﬁcantly outperform the next best model (DTNN), producing less than half of its mean absolute error (MAE).
Errors of other methods are at least one — sometimes two — orders of magnitude greater.
In part, this performance discrepancy might be explained by the hightened suitability of our scattering transform for environments with somewhat limited training-data availability.
Here we speculate that the additional performance gap might be explained by the fact that our graph shift operator ∆ carries the same information as the Coulomb matrix (a proven molecular graph descriptor in itself [32]).
Additionally, our ﬁlters being inﬁnite series’ in powers of the underlying normal operator allows for rapid dispersion of information across underlying molecular graphs, as opposed to e.g. the ﬁlters in GraphConv
[Table]
or SchNet, which do not incorporate such higher powers.
To quantify the effect of including secondorder scattering coefﬁcients, we also include the result of performing kernel-regression solely onﬁrst order features generated through Architecture II of Fig.
2 (GGST + EK).
While results are stillbetter than those of all but one leading approach, incorporating higher order scattering improvesperformance signiﬁcantly.
Discussion of Results in Context
Leaving behind the traditional reliance on graph wavelets, we developed a theoretically well foundedframework for the design and analysis of (generalized) graph scattering networks; allowing forvarying branching rations, non-linearities and ﬁlter banks.
We provided spectrum independentstability guarantees, covering changes in input signals and for the ﬁrst time also arbitrary normalperturbations in the underlying graph-shift-operators.
After introducing a new framework to quantifyvertex-set non-preserving changes in graph domains, we obtained spectrum-independent stabilityguarantees for this setting too.
We provided conditions for energy decay and discussed implicationsfor truncation stability.
Then we introduced a new method of graph-level feature aggregation andextended scattering networks to higher order input data.
Our numerical experiments showed thata simple scattering transform conforming to our framework is able to outperform the traditionalgraph-wavelet based approach to graph scattering in social network graph classiﬁcation tasks.
Oncomplex datasets our method is also competitive with current fully trainable methods, ouperformingall competitors on REDDIT- 12 K. Additionally, higher order graph scattering transforms signiﬁcantlyoutperform current leading graph-based learning methods in predicting atomization energies on QM 7 .
A reasonable critique of scattering networks as tractable models for general graph convolutionalnetworks is their inability to emulate non-tree-structured network topologies.
While transcendingthe wavelet setting has arguably diminished the conceptual gap between the two architectures, thisstructural difference persists.
Additionally we note that despite a provided promising example, it isnot yet clear whether the newly introduced graph-perturbation framework can aptly provide stabilityguarantees to all reasonable coarse-graining procedures.
Exploring this question is ongoing work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] A discussion of how and in which order the main claims
made in Abstract and Introduction were substantiated within the paper is a main focus
of Section
(b) Did you describe the limitations of your work?
[Yes] This is a second focus of Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] This is
part of Section 8.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Every Theorem
and Lemma is stated in a mathematically precise way, with all underlying assumptions
included.
Additionally, Appendix A brieﬂy reviews some terminology that might not
be immediately present in every readers mind, but is utilized in order to be able to state
theoretical results in a precise and concise manner.
(b) Did you include complete proofs of all theoretical results?
[Yes] This is the Focus
of Appendices B, C, D, F, G, H and I. Results in Appendix J
are merely stated, as
statements and corresponding proofs are in complete analogy (in fact almost verbatim
the same) to previously discussed statements and proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Yes; please see
the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] This is the main focus of Appendix K.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Errors for results of conducted experiments are includedin Table 1 and Table 2 respectively.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] This is described at the beginning
of Appendix K
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] All utilized datasets
were matched to the papers that introduced them (cf.
Section K).
Additionally, we
partially built on code corresponding to [12] which we mentioned in Appendix K.
(b) Did you mention the license of the assets?
[Yes] We mentioned in Appendix K that
the
code corresponding to [12] is freely available under an Apache License.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Please see supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[Yes] For the utilized social network datasets,
we
mention in Appendix K that neither personally identifyable data nor content that might
be considered offensive is utilised.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Some Concepts in Linear Algebra
In the interest of self-containedness, we provide a brief review of some concepts from linear algebrautilized in this work that might potentially be considered more advanced.
Presented results are allstandard; a very thorough reference is [25].
Hilbert Spaces: To us, a Hilbert space — often denoted by H — is a vector space over the complexnumbers which also has an inner product — often denoted by x¨ , ¨y H .
Prototypical examples aregiven by the Euclidean spaces C d with inner product x x, y y C d : “ ř di “ 1 x i y i .
Associated to an innerproduct is a norm, denoted by } ¨ } H and deﬁned by } x } H : “ a x x, x y H for x P H .
Direct Sums of Spaces: Given two potentially different Hilbert spaces H and p H , one can formtheir direct sum H ‘ p H .
Elements of H ‘ p H are vectors of the form p a, b q , with a P H and b P p H .
Addition and scalar multiplication are deﬁned in the obvious way byp a, b q ` λ p c, d q : “ p a ` λc, b ` λd q for a, c P H , b, d P p H and λ P C .
The inner product on the direct sum is deﬁned byxp a, b q , p c, d qy H ‘ p H : “ x a, c y H ` x b, d y p H .
As is readily checked, this implies that the norm } ¨ } H ‘ p H on the direct sum is given by}p
[Equation]
Standard examples of direct sums are again the Euclidean spaces, where one has C d “ C n ‘ C m ifm ` n “ d , as is easily checked.
One might also consider direct sums with more than two summands,writing C d “ ‘ di “ 1 C for example.
In fact, one might also consider inﬁnite sums of Hilbert spaces:
The space ‘ 8 i “ 1 H i is made up of those elements a “ p a 1 , a 2 , a 3 , ...
q with a i P H i for which thenorm
[Equation]
is ﬁnite.
This means for example that the vector p 1 , 0 , 0 , 0 , ...
q is in ‘ 8 i “ 1 C , while p 1 , 1 , 1 , 1 , ...
q isnot.
Direct Sums of Maps: Suppose we have two collections of Hilbert spaces t H
i u Γ i “ 1 , t r H
i u Γ
i “ 1 withΓ P N or Γ “ 8 .
Suppose further that for each i ď Γ (resp.
i ă Γ ) we have a (not necessarily linear)map J i : H
i Ñ r H i .
Then the collection t J i u Γ i “ 1 of these ’component’ maps induce a ’composite’map
[Equation]
between the direct sums.
Its value on an element a “ p a 1 , a 2 , a 3 , ...
q P ‘ Γ i “ 1 H i is deﬁned by
[Equation]
[Equation]
Strictly speaking, one has to be a bit more careful in the case where Γ “ 8 to ensure that} J p a q} ‘ 8 i “ 1 r H i ‰ 8 .
This can however be ensured if we have } J i p a
Operator Norm: Let J : H Ñ r H be a linear operator between Hilbert spaces.
We measure its’size’ by what is called the operator norm, denoted by } ¨ } op and deﬁned by
[Equation]
Adjoint Operators Let J : H Ñ r H be a linear operator from the Hilbert space H to the Hilbertspace r H .
Its adjoint J ˚ : r H Ñ H is an operator mapping in the opposite direction.
It is uniquelydetermined by demanding that
[Equation]
holds true for arbitrary f P H and u P r H .
Normal Operators:
If a linear operator ∆ : H Ñ H maps from and to the same Hilbert space,we can compare it directly with its adjoint.
If ∆∆ ˚ “ ∆ ˚ ∆ , we say that the operator ∆ is normal.
Special instances of normal operators are self-adjoint operators, for which we have the strongerproperty ∆ “ ∆ ˚ .
If an operator is normal, there are unitary maps U : H Ñ H diagonalizing ∆ as
[Equation]
with eigenvalues in C .
We call the collection of eigenvalues the spectrum σ p ∆ q of ∆ .
If dim H “ d ,we may write σ p ∆ q “ t λ u di “ 1 .
It is a standard exercise to verify that each eigenvalue satisﬁes|
λ i | ď } ∆ } op .
Associated to each eigenvalue is an eigenvector φ i .
The collection of all (normalized)eigenvectors forms an orthonormal basis of H .
We may then write
[Equation]
Resolvent of a (normal) Operator:
Given a normal operator ∆ on some Hilbert space H , we havethat the operator p ∆ ´
z q : H Ñ H is invertible precisely if z ‰ σ p ∆ q .
In this case we write
[Equation]
p
[Equation]
and call this operator the resolvent of ∆ at z .
It can be proved that the norm of the resolvent satisﬁes
[Equation]
where dist p z, σ p ∆ qq denotes the minimal distance between z and any eigenvalue of ∆ .
Functional Calculus:
Given a normal operator ∆ : H Ñ H on a Hilbert space of dimension d anda complex function g : C Ñ C , we can deﬁne another normal operator obtained from applying thefunction g to ∆ by
[Equation]
For example if g p¨q “ | ¨ | , we obtain the absolute value | ∆ | of ∆ by specifying for all f P H that
[Equation]
Similarly we ﬁnd (if z R σ p ∆ q and for f P H )
[Equation]
where we think of the left-hand-side as applying a function to ∆ , while we think of the right-hand-sideas inverting the operator p ∆ ´ z q .
This now allows us to apply tools from complex analysis also tooperators: If a function g is analytic (i.e. can be expanded into a power series), we have
[Equation]
for any circle S Ď C encircling λ by Cauchy’s integral formula.
Thus, if we chose S large enough toencircle the entire spectrum σ p ∆ q , we have
[Equation]
Frobenius Norm:
Given a ﬁnite dimensional Hilbert space H with inner product x¨ , ¨y H , and anorthonormal basis t φ i u di “ 1 , we deﬁne the trace of an operator A : H Ñ H as
[Equation]
It is a standard exercise to show that this is independent of the choice of orthonormal basis.
Theassociated Frobenius inner product on the space of operators is then given as
[Equation]
Hence the Frobenius norm of an operator is determined by
[Equation]
It is a standard exercise to verify that we have } A } op ď } A } F .
Since the trace is independent of thechoice of orthonormal basis, the Frobenius norm is invariant under unitary transformations.
Moreprecisely, if U, V : H Ñ H are unitary, we have
[Equation]
Frobenius norms can be used to transfer Lipschitz continuity properties of complex functions to thesetting of functions applied to normal operators:
Lemma A.1.
Let g : C Ñ C be Lipschitz continuous with Lipschitz constant D g .
This implies
[Equation]
for normal operators X, Y on H .
Proof.
This proof is taken (almost) verbatim from [38].
For an operator A : H Ñ H denote by A ijits matrix representation with respect to the orthonormal basis t φ i u di “ 1 :
[Equation]
We then have
[Equation]
as a quick calculation shows.
Let now U, W be unitary (with respect to the inner product x¨ , ¨y H )operators diagonalizing the normal operators X and Y as
[Equation]
Since the Frobenius norm is invariant under unitary transformations we ﬁnd
[Equation]
B Proof of Theorem 2.
Theorem B.1.
Let ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p
G q be normal.
If the family t g i p¨qu i P I of bounded functionssatisﬁes A ď
ř
i
P
I | g
i p c q| 2 ď B for all c in the spectrum σ p ∆ q , we have ( @ f P (cid:96) 2 p G q )
A } f } 2 (cid:96) 2 p G
[Equation]
ÿ
[Equation]
Proof.
Writing the normalized eigenvalue-eigenvector sequence of ∆ as p λ i , φ i q | G | i “ 1 , we simply note
[Equation]
Now under the assumption, we can estimate the sum in brackets by A from below and by B fromabove.
Then we need only use Bessel’s (in)equality to prove
[Equation]
C Proof of Theorem 4.1
Theorem C.1.
With the notation of Section 3 and setting B 0 “ 1 , we have:
[Equation]
To streamline the argumentation let us ﬁrst introduce some notation:
Notation C.2.
Let us denote paths in Γ N as q : “ p γ N , ..., γ 1 q .
For f P (cid:96) 2 p G q let us write
[Equation]
Proof.
By Deﬁnition, we have
[Equation]
We proceed in two steps:
Our initial goal is to upper bound a n as
[Equation] [Equation]
for b n : “ ř q P Γ n } f q ´ h q } 2 (cid:96) 2 p G n q with b 0 “ } f ´ h } 2 (cid:96) 2 p G q .
To achieve this we note that (5) isequivalent to
[Equation]
which upon unraveling deﬁnitions may be written asÿ
[Equation] [Equation]
To establish (6)
[Equation]
we note
[Equation]
that in the sum over paths of length n
[Equation]
any p q P Γ n can uniquely be writtenas p q “ p γ n , q q , with the path q P Γ n ´ 1 of length p n ´ 1 q determined by
[Equation]
q .
With this we ﬁndÿ
[Equation]
Thus we can rewrite the left hand side of (6) asÿ
[Equation]
The fact that in each layer the function t χ n p¨qu Ť t g γ n p¨qu γ n P Γ n form a generalized frame with upperframe constant B n implies by Theorem 2.1, that we can further bound this as
[Equation]
Using the Lipschitz continuity of ρ n and P n , we arrive at the desired expression (6).
Having established that
[Equation]
holds true, we note that we can establish
[Equation]
arguing similarly as in the case of (6) by using (for f P (cid:96) 2 p G n ´ 1 q )ÿ
[Equation]
n
[Equation]
together with the frame property and Lipschitz continuities.
We then iterate this inequality and recallthat b 0 “ } f ´ h } 2 (cid:96) 2 p G q .
Using the fact that
[Equation]
we ﬁnally ﬁnd
[Equation]
D Proof or Theorem
Theorem D.1.
Let Φ N and r Φ N be two scattering transforms based on the same module sequenceΩ N and operator sequences D N , r D N with the same connecting operators ( P n “ r P n ) in eachlayer.
Assume R ` n , L ` n ď 1 and B n ď B for some B and n ď N .
Assume that the respectivenormal operators satisfy } ∆ n ´ r ∆ n } F ď δ for some δ ą 0 .
Further assume that the functionst g γ n u γ n P Γ n and χ n in each layer are Lipschitz continuous with associated Lipschitz constantssatisfying L 2 χ n ` ř γ n P Γ n L 2 g γn ď D 2 for all n ď N and some D ą 0 .
Then we have
[Equation]
for all f P (cid:96) 2 p G q .
If B ď 1 { 2 , the stability constant improves to
[Equation]
Notation D.2.
Let us denote scattering propagators based on operators ∆ n and connecting operators
P n by U n and scattering propagators based on operators r ∆ n by r U n .
Similarly, to Notation C.2, let usthen write (with q “ p γ N , ..., γ 1 q )
[Equation]
Proof.
By deﬁnition we have
[Equation]
˛ ‚ loooooooooooooooooooooooooooooooooooooooomoooooooooooooooooooooooooooooooooooooooon “
[Equation]
We deﬁne b n : “ ř q P Γ n } f q ´ r f q } 2 (cid:96) 2 p G n q , with b 0 “ } f ´ h } 2 (cid:96) 2 p G q “ 0 and notea n
[Equation]
n
[Equation]
ÿ
[Equation]
[Table]
[Equation]
[Table]
ˆ
[Equation]
[Table]
` ÿ
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
¸ .
Using (with | a ` b | 2 ď 2 p| a | 2 ` | b | 2 q )
[Equation]
and
[Equation]
(c.f. Lemma A.1 ), we ﬁnd
[Equation]
Using L 2 χ n ` ř γ n P Γ n L 2 γ n ď D 2 , we then infer (using the assumption L ` n , R `
[Equation]
Now if B ď 12 , we have
[Equation]
and results of geometric sums leads to the desired bound after summing over n .
Hence let us assume B ą 12 .
Using similar arguments as before, we ﬁnd
[Equation]
Thus we now know
[Equation]
In total we ﬁnd
[Equation]
where we have estimated the sum over p b n ´ 1 ´ b n q by zero from above again.
This establishes theclaim.
Remark D.3.
To see that this also holds for our Architecture I of Fig.
2, we note that the critical stepis establishing that Lemma A.1 also applies to δ 0 and cos , as deﬁned in Section 2.
Here we establishthat
[Equation]
and
[Equation]
Indeed, since ∆ and r ∆ are (possibly) rescaled graph Laplacians on the same graph, the spectralprojections to their lowest lying eigen space, associated to the eigenvalue λ min “ 0 agree.
Denotingthis spectral projection by P , we have
[Equation]
and we can apply Lemma A.1.
Similar considerations apply to δ 0 .
E Prototypical Example illustrating ω - δ Closeness and δ -Unitary
Equivalence
To investigate the example of Figure 3, we label the vertices of the respective graphs as depicted in Figure 6.
We denote the left graph by G and the right graph by r G .
The node-weights on r G are given as r µ i “ 1 for 1 ď i ď 7 , while on G the weights are given as µ i “ 1 for 1 ď i ď 5 while µ 6 “ 2 .
We then consider the respective un-normalized graph Laplacians ∆ : (cid:96) 2 p G q Ñ (cid:96) 2 p G q and r ∆ : (cid:96) 2 p r G q Ñ (cid:96) 2 p r G q , which for a given adjacency matrix W on a graph signal space (cid:96) 2 p G q with node weights t µ i u i is given as
[Equation]
Such operators are positive and hence | ∆ | “ ∆ (similarly for r ∆ ).
We now need to ﬁnd operators J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q and r J : (cid:96)
2 p r G q Ñ (cid:96) 2 p G q satisfying the conditions of Deﬁnition 4.3.
To construct J , we deﬁne a family t ψ i u 6 i “ 1 of vectors on (cid:96) 2 p r G q as
[Equation]
The map J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q is then deﬁned as
Jf
[Equation] [Equation]
f i ψ i , for any f P (cid:96) 2 p G q .
We take r J : (cid:96)
2 p r G q Ñ (cid:96) 2 p G q to be its adjoint ( r J : “ J ˚ ), which determinedexplicitly by
[Equation]
for any u P (cid:96) 2 p r G q We shall now ﬁrst check the conditions for δ -quasi unitary equivalence, which welist again for convenience; now adapted to our current setting:
[Equation]
We ﬁrst note that since r J “ J ˚ , we have }p J ´ r J ˚ q f } (cid:96) 2 p r G q “ 0 .
Next we note
[Equation]
Furthermore we note
[Equation]
p r G q loooooooomoooooooon “ δ ik “ f i and hence } f ´ r JJf } 2 (cid:96) 2 p G q “ 0 .
It remains to control } u ´ J r Ju } 2 (cid:96) 2 p r G q .
We noter
Ju “ p u 1 , u 2 , u 3 , u 4 , u 5 , p u 5 ` u 6 q{ 2 q J and thus
J
[Equation]
Ju
[Equation] [Equation]
u
[Equation]
u
[Equation] [Equation]
[Equation]
Which implies
[Equation]
and thus
[Equation]
We have
[Equation]
Since Ă W 67 “ 1 { δ 2 by assumption, we have
[Equation]
Thus we have proven δ -unitary-equivalence and it remains to establish p´ 1 q - 12 δ closeness.
Combining Proposition 4.4.12.
and Theorem 4.4.15 of [30], instead of bounding }p r RJ ´ JR q f } (cid:96) 2 p r G q ď12 δ } f } (cid:96) 2 p G q directly, we may instead establish that there are operators J 1 : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q ,
[Equation]
(cid:96)
[Equation]
p
[Equation]
G
[Equation]
2 p G q satisfying
[Equation] [Equation]
and
[Equation]
We chose J 1 “ J and determine Ă J 1 by setting (for ( 1 ď i ď 6 ))
[Equation]
Thus (7) is clearly satisﬁed.
For (8) we note that we have
[Equation]
Thus we have
[Equation]
¯ as before.
It remains to establish (9).
We have
[Equation]
while we have
[Equation]
We have (with all node-weights on (cid:96) 2 p G q equal to unity)
[Equation]
˜ÿ
[Equation]
˜ÿ
[Equation]
¸
And thus
[Equation]
which proves the claim.
F Proof of Lemma 4.4
Lemma F.1.
In the setting of Deﬁnition 4.3 let ∆ and r ∆ be ω - δ -close and satisfy } ∆ } op , } r ∆ } op ď
Kfor some K ą 0 .
If g : C Ñ C is holomorphic on the disk B K ` 1 p 0 q of radius p K ` 1 q , there is aconstant C g ě 0 so that
[Equation]
with C g depending on g , ω and K .
Proof.
Without loss of generality, let us assume that K ą | ω | .
Let us denote the circle of radius r in 706 C by S r .
For any holomorphic function g and (normal) operator ∆ whose spectrum is enclosed bythe circle S r , we can express the operator g p ∆ q as
[Equation]
as discussed in Appendix A (see also [7] for more details).
Note that in our case the resolvent
R p z, ∆ q “ p ∆ ´ z q ´ 1 is well deﬁned for | z | ě K , since with our assumptions all eigenvalues arewithin the circle of radius K .
Additionally note that we havedist
[Equation]
if | z | ě K .
The same holds true after replacing ∆ with r ∆ .
Since for any normal operator ∆ we have} R p z, ∆ q} op
[Equation]
p
[Equation]
we ﬁnd
[Equation]
To quantify the difference } R p z, r ∆ q J ´ JR p z, ∆ q} op in terms of the difference } r R p ω q J ´
JR p ω q} op ď δ , we deﬁne the function
[Equation]
for which
[Equation]
holds, as proved (in more general form) in Lemma 4.5.9 in [30].
Since on S K ` 1 we have and| z ´ ω | ď 2 K ` 1 hence γ 0 p z q ď 2 p K ` 1 q , we ﬁnd
[Equation]
Thus we may set
[Equation] [Equation]
G Proof of Theorem 4.5
We state and prove a somewhat more general theorem, incorporating also the case where the identiﬁcation operators only almost commute with connecting operators or non-linearities.
We also wouldlike to point out that the constant 2 in Deﬁnition 4.3 is arbitrary and any constant larger than onewould sufﬁce.
Much more details are provided in Chapter IV of [30].
Theorem G.1.
Let Φ N , r Φ N be scattering transforms based on a common module sequence Ω N anddiffering operator sequences D N , r D N .
Assume R ` n , L ` n ď 1 and B n ď B for some B and n ě 0 .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q , r J n : (cid:96) 2 p r G n q Ñ (cid:96) 2 p G n q( 0 ď n ď N ) so that the respective signal spaces are δ -unitarily equivalent, the respective normaloperators ∆ n , r ∆ n are ω - δ -close as well as bounded (in norm) by K ą 0 and the connectingoperators satisfy } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q ď δ } f } (cid:96) 2 p G n ´ 1 q .
For the common module sequenceΩ N assume that the non-linearities satisfy } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q ď δ } f } (cid:96) 2 p G n q and that theconstants C χ n and t C g γn u γ n P Γ N associated through Lemma 4.4 to the functions of the generalizedframes in each layer satisfy C 2 χ n ` ř
[Equation]
ď D
[Equation]
for some D ą 0 .
Denote the operatorthat the family t J n u n of identiﬁcation operators induce on F N through concatenation by J N :
F N Ñ Ă F N .
Then we have with K N “ a p 8 N ´
qp 2 D 2 ` 12 B q{ 7 ¨ B N ´ 1 if B ą 1 { 8 and
K N “ a p
[Equation]
`
[Equation]
B q ¨ p 1 ´ B N q{p 1 ´ B q if B ď 1 { 8 that
[Equation]
If additionally } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q “ 0 or } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q “ 0 holds ineach layer, then we have K N “ a p 4 N ´
qp 2 D 2 ` 4 B q{ 3 ¨ B N ´ 1 if B ą 1 { 4 and K N “ 739 a p 2 D 2 ` 4 B q ¨ p 1 ´ B N q{p 1 ´ B q if B ď 1 { 4 .
If both additional equations hold, we have
K N “ a p 2 N ´ 1 q 2 D 2 ¨ B N ´ 1 if B ą 1 {
and K N “ a 2 D 2 ¨ p 1 ´ B N q{p 1 ´ B q if B ď 1 { 2 .
Notation G.2.
Let us denote scattering propagators based on operators ∆ n and P n by U n andscattering propagators based on operators r ∆ n and r P n by r U n .
Similarly, to Notation D.2 and , let usthen write (with q “ p γ N , ..., γ 1 q )
[Equation]
Proof.
By deﬁnition we have} J Φ N p f q ´ r
[Equation]
N p
[Equation]
0 f
[Equation] [Equation]
Ă F N “
N ÿ n “¨ ˝ ÿ q P Γ n ´} J n χ n p ∆ n q ρ n p P n p f q qq ´ χ n p r ∆ n q ρ n p P n p r f q qq} 2 (cid:96) 2 p r G n q ˛ ‚ looooooooooooooooooooooooooooooooooooooooomooooooooooooooooooooooooooooooooooooooooon “ : a n .
We deﬁne b n : “ ř q P Γ n } J n f q ´ r f q } 2 (cid:96) 2 p r G n q , with b 0 “ } J 0 f
´ J 0 f } 2 (cid:96) 2 p r G q “ 0 and notea n ` b n “ ÿ q P Γ n ´
[Equation]
f
[Equation]
p
[Equation]
n q `
[Equation]
n
[Equation]
n
[Equation]
p
[Equation]
q ¸ .
Using
[Equation]
where the second inequality arises from permuting the identiﬁcation operator J n through non-linearityand connecting operator.
Using C 2 χ n ` ř
[Equation]
n C 2 γ n ď D 2 , we then infer
[Equation]
If B ď 18 , summing over n and using a geometric sum argument yields the desired stability constant.
Hence let us assume B ą 18 .
Using similar arguments as before, we ﬁnd
[Equation]
In total we ﬁnd
[Equation]
If one of the additional equations holds, we ﬁnd
[Equation]
and
[Equation]
Arguing as previously yields the desired stability bounds.
If both additional equations are satisﬁed the proof is virtually the same as the one for Theorem4.2.
H Details on Energy Decay and Truncation Stability
We ﬁrst prove the statement made about the relation between truncation stability and energy:
Lemma H.1.
Given the energy W N : “ ř p γ N ,...,γ 1 qP
[Equation]
N
} U r γ N s ˝ ...
˝
U r γ 1 sp f q}
(cid:96) 2 p G N q stored inthe network at layer N , we have after extending Φ N p f q by zero to match dimensions with Φ N ` 1 p f qthat
[Equation]
Proof.
We note
[Equation]
In fact one can prove even more:
Lemma H.2.
The energy W N stored in layer N satisﬁes
[Equation]
with constants
[Equation]
Proof.
min
[Equation]
and similarly
[Equation]
This yields the starting point for our induction.
Now for the inductive step assume the claim holds upuntil layer N ´ 1 .
Then we have
[Equation]
using Notation C.2.
We note
N
[Equation]
Every path r q P Γ N may be written as q “ p γ n , q q , for some γ n P Γ n and q P Γ N ´ 1 .
Thus we haveÿ
[Equation]
Inserting this in the above equation yields
N ÿ n
[Equation]
¸ looooooooooooooooooooooooooooooooooooooooooooooooomooooooooooooooooooooooooooooooooooooooooooooooooon
[Equation]
We have
[Equation]
by the operator frame property.
With this we ﬁnd:
[Equation]
after unravelling the deﬁnition
[Equation]
The induction hypothesis together with the deﬁnition of C ˘ N now yields the claim.
With this we now prove our main theorem concerning energy decay.
Theorem H.3.
Let Φ 8 be a generalized graph scattering transform based on a an operator sequence 781 D 8 “ p P n , ∆ n q 8 n “ 1 and a module sequence Ω 8 with each ρ n p¨q ě 0 .
Assume in each layern ě 1 that there is an eigenvector ψ n of ∆ n with solely positive entries; denote the smallestentry by m n : “ min i P G n ψ n r
i s .
Denote the eigenvalue corresponding to ψ n by λ n .
Quantify the
’spectral-gap’ opened up at this eigenvalue through neglecting the output-generating function byη n : “
[Equation]
p λ n q| 2 and assume B n m n ě η n .
We then have
[Equation]
Proof.
Denote the spectral projection (i.e. the orthogonal projection projecting to the space ofeigenvectors) onto the eigenspace corresponding to λ n by P nc .
Then we have
[Equation]
By orthogonality of the spectral projection, we then have
[Equation]
Furthermore, we have
[Equation]
with equality if the multiplicity of λ N is exactly one.
With this we ﬁnd
[Equation]
Since the image of ρ N is contained in R ` by assumption
[Equation]
we have
[Equation]
Here the second to last inequality follows since in any ﬁnite dimensional vector space, the 1 -norm islarger than the 2 -norm (
|| f ||
ě
||
f
|| 2 ) and all weights are assumed to satisfy µ i ě 1 .
Thus we nowknow
[Equation]
Inserting this in our estimate for W N p f q we ﬁnd
[Equation]
Taking N to inﬁnity, we know that C ` N converges to something larger than zero by assumption.
For products of the form N ś n “ 0 p 1 ´ q n q with 0 ď q n ă 1 it is a standard exercise to prove that the limitis non-zero precisely if the sum over the q n converges.
Combining the above result with Lemma H.2,we obtain as an immediate Corollary:
Corollary H.4.
In the setting of Theorem 4.6, the generalized scattering transform satisﬁes Φ ´ 1 8 p 0 q “t 0 u if C ˘ N Ñ C ˘ for some positive constants C ˘ and ř Nn “ 1 p m n ´ η n { B n q Ñ 8 as N Ñ 8 .
I Stability of Graph Level Feature Aggregation
[Figure]
I.1 General non-linear feature aggregation:
[Figure]
Our main stability theorem for non-linear feature aggregation is as follows:
[Figure]
Theorem I.1.
We have
[Figure]
[Equation]
With the conditions and notation of Theorem 4.2 we have
[Equation]
Additionally, in the setting of Theorem 4.5, assuming that for each n ď N the identiﬁcation operator
J n satisﬁes
[Equation]
( 2 ď k ď p n ) implies ( @ f P (cid:96) 2 p G q )
[Equation]
Furhermore, under the assumptions of Corollary H.4 Ψ 8 p f q “ 0 implies f “ 0 .
Proof.
Let f, h P (cid:96) 2 p G q .
To prove the ﬁrst two claims, it sufﬁces to prove
[Equation]
, and
[Equation]
Both statements follow immediately, as soon as we have proved
[Equation]
(cid:96)
[Equation]
for arbitrary choices of p and G .
To this end we note that for p
ě 2 we have } f } (cid:96)
p p G q ď } f } (cid:96)
[Equation]
qby the monotonicity of p -norms, while we have } f }
[Equation]
1 p G q ď ?
µ G ¨ } f }
[Equation]
G q by Hölder’s inequality.
With this we ﬁnd
[Equation]
where we have employed the reverse triangle inequality in the ﬁrst step.
To prove the second claim, we note that we have
[Equation]
Thus it remains to bound the last expression.
We have
[Equation]
By our results of Appendix C and since we assume admissibility, we have
[Equation]
Thus in total
[Equation]
from which our stability claim follows.
It remains to prove that the assumptions of Corollary H.4 Ψ 8 p
f q “ 0 imply f “ 0 .
But since
N Gp p f q “ 0 implies f “ 0 , this is clear.
I.2 Low-Pass feature Aggregation
The main assumption we have in this section is that each operator ∆ n (and r ∆ n ) has a simple lowestlying eigenvalue equal to zero.
We denote the associated eigenvector (determined up to a complexphase) by ψ ∆ n and the associated spectral projection to the lowest lying eigenvalue by P ∆ n .
It actsas
[Equation]
Now we are ready to state our main stability result under these circumstances:
Theorem I.2.
We have
[Equation]
With the conditions and notation of Theorem 4.2 and under the additional assumption }p P ∆ n ´
P r ∆ n q} op ď K ¨ δ for n ď N and some K ě 0 , we have
[Equation]
In the setting of Theorem 4.5 and under the additional assumption |} P ∆ n f } (cid:96)
[Equation]
G
[Equation]
q ´}
[Equation] [Equation]
q | ď
Kδ || f || (cid:96) 2 p
G n q for all f P (cid:96) 2 p G n q ( n ď N ), we have
[Equation]
Proof.
Let f, h P (cid:96) 2 p G q .
To prove the ﬁrst claim, it sufﬁces to prove
[Equation]
Ψ
[Equation]
N
[Equation]
This immediately follows from the fact that for all f P (cid:96) 2 p G n q
[Equation]
n q by Hölder’s inequality.
The next claim we want to prove is that we have for all f P (cid:96) 2 p G q
[Equation]
We note
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
˛ ‚
Hence we need to bound the expression " }
[Equation]
We note
[Equation]
and thus
[Equation]
and the claim follows.
Finally we want to prove
[Equation]
[Table]
[Equation]
We note
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
which proves the claim.
In establishing triviality of the ’kernel’, we have to be a tiny bit more careful:
Theorem I.3.
In the setting of of Corollary H.4, assume that in each layer n , the output generatingfunction χ n of the underlying scattering transform satisﬁes χ n p 0 q ‰ 0 and χ n p λ i q “ 0 for orderednon-zero eigenvalues λ 2 ď ... ď λ | G n | of the operator ∆ n .
Then Ψ |x¨ , ¨y|p f q “ 0 implies f “ 0 .
Proof.
Under these assumptions, we do not lose any information by projecting to ψ ∆ n in each(cid:96) 2 p G n q , since the image of χ n p ∆ n q is already contained in the one-dimensional space generated bythe lowest lying eigenvector ψ ∆ n .
J Details on Higher Order Scattering
Node signals capture information about nodes in isolation.
However, one might also want to analyseor incorporate information about binary, ternary or even higher order relations between nodes, suchas distances or angles between nodes representing atoms in a molecule.
This can be formalized byconsidering tensorial input signals:
Tensorial input: A 2 -tensor on a graph G , as it was already utilized in Section 6, is simply anelement of C | G |ˆ| G | or – equivalently – a map from G ˆ G to C , since it associates a complex numberto each element p g 1 , g 2 q P G ˆ G .
Since G ˆ G is precisely the set of (possible) edges E , we canequivalently think of 2 -tensors edge-signals.
A 3 -tensor an element of C | G |ˆ| G |ˆ| G | or equivalentlya map from G ˆ G ˆ G ” G 3 to C .
A 4 -tensor then is a map from G 4 ” G ˆ G ˆ G ˆ G to Cor equivalenlty an element of C | G |ˆ| G |ˆ| G |ˆ| G | and so forth.
Clearly the space of k -tensors forms alinear vector space.
Addition and scalar multiplication by λ P C are given by
[Equation]
with f and g being k -tensors.
For ﬁxed k , we equip the space of k -tensors with an inner productaccording to
[Equation] [Equation]
and denote the resulting inner-product space by (cid:96) 2 p G k q .
Operators on Spaces of Tensors: Since for ﬁxed k the space (cid:96) 2 p G k q is simply a | G | k -dimensionalcomplex inner product space, there are exist normal operators ∆ k : (cid:96) 2 p G k q Ñ (cid:96) 2 p G k q on this space.
Note that the k in ∆ k signiﬁes on which space this operator acts.
It does not signify that an operatoris raised to the k th power.
Setting for example node-weights µ i and edge weights µ ik to one, theadjacency matrix W as well as normalized or un-normalized graph Laplacians constitute self-adjointoperators on (cid:96) 2 p G 2 q , where they act by matrix multiplication.
Higher order Scattering Transforms: We can then follow the recipe laid out Section 3 in constructing k th -order scattering transforms; all that we need are a module sequence Ω N and an operatorsequence D kN : “ p P kn , ∆ kn q Nn “ 1 , where now P kn : (cid:96) 2 p G kn ´
1 q Ñ (cid:96) 2 p G kn q and ∆ kn : (cid:96) 2 p
G kn q Ñ (cid:96) 2 p
G kn q .
To our initial signal f P (cid:96) 2 p G k q we ﬁrst apply the connecting operator P k 1 , yielding a signal representation in (cid:96) 2 p G k 1 q .
Subsequently, we apply the pointwise non-linearity ρ 1 .
Then we apply our graph ﬁlters t χ 1 p ∆ k 1 qu Ť t g γ 1 p ∆ k 1 qu γ 1 P
[Equation]
to ρ 1 p P k 1 p f qq yielding the output V 1 p f q : “ χ 1 p ∆ k 1 q ρ 1 p P k 1 p f qq as well as the intermediate hidden representations t U 1 r γ 1 sp f q : “ g γ 1 p ∆ k 1 q ρ 1 p P k 1 p f qqu γ 1 P Γ 1 obtained in the ﬁrst layer.
Here we have introduced the one-step scattering propagator U n r γ n s : (cid:96) 2 p G kn ´ 1 q Ñ (cid:96) 2 p G kn q mapping f ÞÑ g γ n p ∆ n q ρ n p P n p f qq as well as the output generating operator
V n : (cid:96) 2 p
G kn ´ 1 q Ñ (cid:96) 2 p
G kn q mapping f to χ n p ∆ kn
q ρ n p P kn p f qq .
Upon deﬁning the set Γ N ´ 1 : “ Γ N ´ 1 ˆ ... ˆ Γ 1 of paths of length p N ´ 1 q terminating in layer N ´ 1 (with Γtaken to be the one-element set) and iterating the above procedure, we see that the outputs generated in the N th -layer are indexed by paths Γ N ´terminating in the previous layer.
We denote the resulting feature map by Φ kN and write F kN for the corresponding feature space.
Thenode-level stability results of the preceding sections then readily translate to higher order scatteringtransforms.
As the respective proofs are identical to the corresponding results for the node setting, we do notrepeat them here.
Theorem J.1.
With the notation of Section 4, we have for all f, h P (cid:96) 2 p G k q :
[Equation]
Theorem J.2.
Let Φ N and r Φ N be two scattering transforms based on the same module sequenceΩ N and operator sequences D kN , r D kN with the same connecting operators ( P kn “ r P kn ) in eachlayer.
Assume R ` n , L ` n ď 1 and B n ď B for some B and n ď N .
Assume that the respectivenormal operators satisfy } ∆ kn ´ r ∆ kn } F ď δ for some δ ą 0 .
Further assume that the functionst g γ n u γ n P Γ n and χ n in each layer are Lipschitz continuous with associated Lipschitz constantssatisfying L 2 χ n ` ř γ n P Γ n L 2 g γn ď D 2 for all n ď N and some D ą 0 .
Then we have for all
[Equation]
P
[Equation]
q
[Equation]
Theorem J.3.
Let Φ kN , r Φ kN be higher order scattering transforms based on a common modulesequence Ω N and differing operator sequences D kN , r D kN .
Assume R ` n , L ` n ď 1 and B n ď Bfor some B and n ě 0 .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G kn q Ñ (cid:96) 2 p r G kn q ,r J n : (cid:96) 2 p r G kn q Ñ (cid:96) 2 p G kn q ( 0 ď n ď N ) so that the respective signal spaces are δ -unitarily equivalent,the respective normal operators ∆ kn , r ∆ kn are ω - δ -close as well as bounded (in norm) by K ą 0 and theconnecting operators satisfy } r P kn J n ´ 1 f ´ J n P kn f } (cid:96) 2 p r G kn q ď δ } f } (cid:96) 2 p G kn ´ 1 q .
For the common modulesequence Ω N assume that the non-linearities satisfy } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G k n q ď δ } f } (cid:96) 2 p G k n qand that the constants C χ n and t C g γn u γ n P Γ N associated through Lemma 4.4 to the functions of thegeneralized frames in each layer satisfy C 2 χ n ` ř
γ n P Γ N C 2 g γn ď D 2 for some D ą 0 .
Denote theoperator that the family t J n u n of identiﬁcation operators induce on F kN through concatenation by
J N : F kN Ñ Ă F kN .
Then we have with K N “ a p 8 N ´
qp 2 D 2 ` 12 B q{ 7 ¨ B N ´ 1 if B ą 1 { 8 and
K
[Equation]
a
[Equation]
q
[Equation]
ď
[Equation]
8 that
[Equation]
If additionally } r P
[Equation]
holds ineach layer, then we have K N “ a
[Equation] [Equation]
4 and K N
[Equation]
N q{p
[Equation]
q if B ď 1 { 4 .
If both additional equations hold, we have
K N “
[Equation]
ą
[Equation]
2 and K N “
[Equation]
´
[Equation]
q if B ď 1 { 2 .
The map N Gp introduced in (4) can also be adapted to aggregate higher-order tensorial features intograph level features: With
[Equation]
q :
[Equation]
and µ G k : “ ř
[Equation]
G |
[Equation] [Equation]
µ
[Equation]
1 ,...,i k , we deﬁne
[Equation]
Given a feature map Φ kN with feature space
[Equation]
we obtain a corresponding map Ψ kN mapping from (cid:96) 2 p G k q to
[Equation]
by concatenating Φ kN with the map that the family of non-linear maps t N p n G kn u Nn “ 1 induces on F N byconcatenation.
The resulting map Ψ kN again has stability properties analogous to the node level case:
Theorem J.4.
Assuming admissibility, we have
[Equation]
for all f, h P (cid:96) 2 p G q .
With the conditions and notation of Theorem J.2 we have
[Equation]
Additionally, in the setting of Theorem J.3, assuming that for each n ď N the identiﬁcation operator
J n satisﬁes
[Equation]
for 2 ď r ď p n implies ( @ f P (cid:96) 2 p G k q )}
[Equation]
As the proofs here are virtually the same as for the corresponding results in previous sections –essentially only replacing G by G k , we omit a repetition of them here.
K Additional Details on Experiments
Here we provide additional details on utilized scattering architectures, training procedures, datasetsand (performance of) other methods our approach is being compared to.
Irrespective of task, ourmodels are trained on an NVIDIA DGX A100 architecture utilizing between two and eight NVIDIA
Tesla A100 GPUs with 80GB memory each.
Running 10 -fold cross validation for the respectiveexperiments took at most 71 hours (which was needed for social network graph classiﬁcation on
REDDIT- 12 K).
K.1 Social Network Graph Classiﬁcation
Datasets: The data we are working with is taken from [45].
In particular this work introduced sixsocial network datasets extracted from from scientiﬁc collaborations (COLLAB), movie collaborations (IMDB-B, IMDB-M) and Reddit discussion threads (REDDIT-B, REDDIT- 5 K, REDDIT- 12 K).
Data is anonymised and contains no content that might be considered offensive.
Each graph carries aclass label, and the goal is to predict this label.
Some basic properties of these datasets are listed in
Table 3 below.
[Table]
These datasets contain graph structures, however they don’t contain associated weights or graphsignals.
Having unspeciﬁed weights simply means that the adjacency matrix W from which we
[Table]
[Equation]
on which our operator ∆ is based simply has each entry corresponding to an edge set to unity.
Ifno edge is present between vertices i and j , the entry W ij is set to zero.
It remains to solve theproblem of the missing input signals.
Our strategy is to generate signals reﬂecting the geometryof the underlying graph.
We do this by utilizing features that associate to each node a number thatcharacterizes its role or importance within its local environment or within the entire graph.
We brieﬂydescribe them here:
Degree : The degree of a node is the number of edges incident at this node.
Eccentricity : For a connected graph, the eccentricity of a node is the maximum distance
from this node to all other nodes.
On a disconnected graph it is not deﬁned.
Clustering :
For unweighted graphs the clustering c p u q of a node u is the fraction of possible
triangles through that node that actually exist.
It is calculated as
[Equation]
Number of triangles : The number of triangles containing the given node as a vertex.
Core number : A k-core is a maximal subgraph that only contains nodes of degree k or
more.
The core number of a node is the largest value k of a k-core containing that node.
Clique number :
A clique is a subset of vertices of an undirected graph such that every two
distinct vertices in the clique are adjacent.
This input assigns the number of cliques thenodes participates in to each node.
Pagerank :
This returns the PageRank of the respective nodes in the graph.
PageRank
computes a ranking of the nodes in the graph based on the structure of the edges.
Originally
it was designed as an algorithm to rank web pages.
For the ﬁrst three datasets listed in Table 3 we utilize all listed input features.
For the latter threedatasets we have to refrain from using eccentricity as an input signal, as these datasets contain graphsthat have multiple non-connected graph components.
Scattering Architecture: We chose a generalized scattering architecture of depth N “ 4 .
Asnormal-operators, we utilize in each layer the un-normalized graph Laplacian L “ D ´ W scaledby its largest eigenvalue ( ∆ “ L { λ max p L q ).
Filters are chosen as 12 p sin p π {
[Equation]
ψ ∆ ψ J ∆ s , sin p π ¨ ∆ q , r cos p π ¨ ∆ q ´ ψ ∆ ψ J ∆ sq , which allows to specify the output generating functionsolely by demanding χ p 0 q “ 1 and χ p λ q “ 0 on all other eigenvalues of ∆ .
Here ψ ∆ is thenormalized vector of all ones (satisfying ∆ ψ ∆ “ 0 ).
Connecting operators are chosen as the identity,while we set ρ n ě 1 p¨q “ | ¨ | .
We note that for connected graphs, this recovers Architecture I of Fig.
On disconnected graphs (as they can appear in the REDDIT datasets), we however do not accountfor vectors other than ψ ∆ in the lowest-lying eigenspace of the graph Laplacian.
This scatteringarchitecture is then applied to each of these input signal individually.
For each input signal, thisreturns a feature vector with 1 ` 4 ` 16 ` 64 “ 85 entries.
These individual feature vectors arethen concatenated into one ﬁnal composite feature vector for each graph.
Concerning applicabletheoretical results, we note the following:
Training Procedure: We train RBF kernel support vector classiﬁers on our composite scatteringfeatures.
We ﬁx (cid:15) “ 0 . 1 .
The hyperparameter γ scaling the exponent is chosen from
G pool : “ t 0 .
1 , 1 , 10 , 100 u , while we pick the C that controls the error our slack variables introduce among
C pool : “ t 0 .
001 , 0 . 01 , 0 . 1 , 1 , 10 , 25 , 50 , 100 , 1000 u .
We chose these parameters in agreement with the choices of [12] to facilitate comparison betweenthe two works.
We could simply implement the training of the RBF-classiﬁer on our composite scattering featuresby dividing each social-network dataset into 10 folds, then iteratively choosing one fold for testingand among the remaining 9 folds randomly choosing one for validation (i.e. for tuning the hyperparameters).
Instead, following [12] (whose code is released under an Apache license and on whichwe partially built), we take a slightly different approach: We still randomly split our dataset into 10folds.
Among the 10 folds, we iteratively pick one for testing.
Say we have picked the n t h fold fortesting.
Then there are 9 remaining folds.
We iteratively pick the m t hn (with 1 ď m n ď 9 ) of theremaining 9 folds for choosing hyperparameters.
This leaves 8 folds on which we train our modelfor each choice of hyper parameter in C pool ˆ G pool .
The resulting classiﬁers are all evaluated on them t hn fold.
The one that performs best is retained as classiﬁer m n .
As m n varies between 1 and 9(still for ﬁxed n ), this yields a set t f m n : 1 ď m n ď 9 u of nine classiﬁers.
From these we build theclassiﬁer f n , whose classiﬁcation result is obtained from a majority vote among the nine classiﬁersin t f m n : 1 ď m n ď 9 u .
Then we evaluate the performance of f n on the n t h fold to obtain then t h estimation of how well our model performs.
As n varies from one to ten, we built the meanand variance of the performances of the classiﬁers f n on the n t h fold expressed as the percentage ofcorrect classiﬁcations.
Reference Methods: To allow for a comparison of our results to the literature, typical classiﬁcationaccuracies for graph algorithms on social network datasets are displayed in Table 1.
Following thestandard format of reporting classiﬁcation accuracies, they are presented in the format (Accuracy ˘standard deviation).
If results are not reported for a dataset, we denote this as not available (N/A).
The ﬁrst three rows of Table 1 display results for graph kernel methods; namely Weisfeiler-Lehmangraph kernels (WL, [34]), Graphlet kernels (Graphlet, [35]) and deep graph kernels (DGK, [44]).
Thesubsequent rows display results for geometric deep learning algorithms: Deep graph convolutional neural networks (DGCNN,[
), Patchy-san (PSCN (with k=10), [
), recurrent neural network autoencoders (S2S-N2N-PP, [16]) and graph isomorphism networks (GIN [43]).
These results aretaken from [
Additionally we compare with P-Poinc [20], which embeds nodes into a hyperbolic space (the Poincare ball, to be precise)
, GSN-e [
] which combines message passing with structural features extracted via subgraph isomorphism and WKPI kC [ which utilizes a weighted kernelwithin its metric learning framework.
The second to last row (
GS
SVM [ provides a result that isalso based on a method that combines a static scattering architecture with a support vector machine.
Its ﬁlters are based on graph wavelets built from differences between lazy random walks that have propagated at different time scales.
K.2 Regression of Quantum Chemical Energies
Dataset: The dataset we consider is the QM 7 dataset, introduced in [2, 32].
This dataset containsdescriptions of 7165 organic molecules, each with up to seven heavy atoms, with all non-hydrogen atoms being considered heavy.
A molecule is represented by its Coulomb matrix C Clmb , whoseoff-diagonal elements
[Equation]
(10) correspond to the Coulomb-repulsion between atoms i and j , while diagonal elements encode apolynomial ﬁt of atomic energies to nuclear charge [32]:
[Equation]
For each atom in any given molecular graph, the individual Cartesian coordinates R i and the atomiccharge Z i are also accessible individually.
To each molecule an atomization energy - calculated viadensity functional theory - is associated.
The objective is to predict this quantity, the performancemetric is mean absolute error.
Numerically, atomization energies are negative numbers in the range´ 600 to ´ 2200 .
The associated unit is r kcal/mol s .
Scattering Architecture: Off-diagonal entries in the Coulomb Matrix clearly represent an inversedistance.
A weight of zero can then heuristically be thought of as the inverse distance between twoinﬁnitely separated atoms.
After calculating the degree matrix D associated to C , we obtain thegraph Laplacian once more as L “ D ´ C and set our normal operator to
[Equation]
If we continuously vary the distances in (10), staying clear of zero, then the adjacency matrixand hence the graph Laplacian L varies continuously.
As long as we avoid complete degeneracy,the largest eigenvalue λ max p L q will remain positive.
This implies that our normal operator ∆varies continuously under changes of the inter-atomic distances, which implies that our featurevector also varies continuously, as distances are changed.
Connecting operators are set to theidentity, while non-linearities are ﬁxed to ρ n ě 1 p¨q “ | ¨ | .
Filters are chosen as p sin p π { 2 ¨ ∆ q ,cos p π { 2 ¨ ∆ q , sin p π ¨ ∆ q , cos p π ¨ ∆ qq acting through matrix multiplication.
The output generatingfunctions are set to the identity as well.
Graph level features are aggregated via the map N E 5 p¨qof Section 6; slightly modiﬁed to neglect the normalizing factor in the ﬁrst entry for improvedconvenience in numerical implementability.
As weights µ ij for our second-order feature space are setto unity and molecular graphs in QM 7 contain at most 23 -molecules, we note that ?
µ G 2 ď ?
Going through the proofs of our graph-level stability results, we see that they remain valid aftermultiplying each stability constant by 23 .
The Coulomb matrix (divided by a factor of 10 as thisempirically improved performance) is then also utilized as an edge level input signal.
Node levelfeatures are obtained by applying the above architecture to the node level information provided bythe respective atomic charges t Z i u on each graph.
We aggregate to graph level features using N G 5(cf.
Section 5), again neglecting the normalizing factor in the ﬁrst entry for improved convenience inimplementing.
The network depth is set to N “ 4 in both cases.
We then concatenate graph levelfeatures obtained from node- and edge level input into a composite scattering feature vector.
Training Procedure: The QM 7 dataset comes with a precomputed partition into ﬁve subsets; eachcontaining a representative amount of heavy and light molecules covering the entire complexity rangeof QM 7 .
To allow for 10 -fold cross validation, we further dissect each of these subsets into twosmaller datasets, one containing graphs indexed by an even number, one containing graphs indexedby an odd number.
On these 10 -subsets, we then perform 10-fold cross validation.
Among the 10folds, we iteratively pick one for testing.
Say we have picked the n t h fold for testing.
Then thereare 9 remaining folds.
We iteratively pick the m t hn (with 1 ď m n ď 9 ) of the remaining 9 folds forchoosing hyperparameters.
This leaves 8 folds on which we train our model for each choice of hyperparameter in C pool ˆ G pool .
This yields 8 regression models, which we average to built our ﬁnalpredictor for the n t h run.
This mean absolute error of this predictor is then evaluated on the n t h foldwhich was retained for testing.
As n varies from one to ten, we built the mean and variance of theperformances of the generated regression models.
We chose log -linear equidistant hyperparametersfrom
G pool : “ t 0 .
03 , 0 . 3 , 3 , 30 u , and
[Table]
Reference Methods: We comprehensively evaluate our method against 11 popular baselines andstate of the art approaches.
Among these methods are graph convolutional methods such as GraphConv[18], Weave [17] or SchNet [33].
MPNN [13] and its variant DMPNN [46] are models consideringedge features during message passing.
AttentiveFP [42] is an extension of the graph attentionframework, while N-Gram [22] is a pretrained method.
Results for these methods as well as for
GROVER are taken from [31].
PhysChem [47] learns molecular representations via fusing physicaland chemical information.
Deep Tensor Neural Networks (DTNN [41]) are adaptable extensions of the
Coulomb Matrix featurizer mapping atom numbers to trainable embeddings which are then updatedbased on distance information and other (node-level) atomic features.
Finally Path-Augmented Graph
Transformer Networks (PATGN, [6]) exploit the connectivity structure of the data in a global attentionmechanism.