L IMITLESS STABILITY FOR G RAPH C ONVOLUTIONAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
This work establishes rigorous, novel and widely applicable stability guarantees and transferability bounds for graph convolutional networks – without reference to any underlying limit object or statistical distribution.
Crucially, utilized graphshift operators (GSOs) are not necessarily assumed to be normal, allowing for the treatment of networks on both directed- and for the ﬁrst time also undirected graphs.
Stability to node-level perturbations is related to an ’adequate (spectral) covering’ property of the ﬁlters in each layer.
Stability to edge-level perturbations is related to Lipschitz constants and newly introduced semi-norms of ﬁlters.
Results on stability to topological perturbations are obtained through recently developed mathematicalphysics based tools.
As an important and novel example, it is showcased that graph convolutional networks are stable under graph-coarse-graining procedures (replacing strongly-connected sub-graphs by single nodes) precisely if the GSO is the graph Laplacian and ﬁlters are regular at inﬁnity.
These new theoretical results are supported by corresponding numerical investigations.
I NTRODUCTION
Graph Convolutional Networks (GCNs) (Kipf & Welling, 2017; Hammond et al., 2011; Defferrard et al., 2016) generalize Euclidean convolutional networks to the graph setting by replacing convolutional ﬁlters by functional calculus ﬁlters; i.e. scalar functions applied to a suitably chosen graph-shift-oprator capturing the geometry of the underlying graph.
A key concept in trying to understand the underlying reasons for the superior numerical performance of such networks on graph learning tasks (as well as a guiding principle for the design of new architectures) is the concept of stability .
In the Euclidean setting, investigating stability essentially amounts to exploring the variation of the output of a network under non-trivial changes of its input (Mallat, 2012; Wiatowski & Bölcskei, 2018).
In the graph-setting, additional complications are introduced: Not only input signals, but now also the graph shift operators facilitating the convolutions on the graphs may vary.
Even worse, there might also occur changes in the topology or vertex sets of the investigated graphs – e.g. when two dissimilar graphs describe the same underlying phenomenon – under which graph convolutional networks should also remain stable.
This last stability property is often also referred to as transferability (Levie et al., 2019a).
Previous works investigated stability under changes in graph-shift operators for speciﬁc ﬁlters (Levie et al., 2019b; Gama et al., 2020) or the effect of graph-rewiring when choosing a speciﬁc graph shift operator (Kenlay et al., 2021).
Stability to topological perturbations has been established for (large) graphs discretising the same underlying topological space (Levie et al., 2019a), the same graphon (Ruiz et al., 2020; Maskey et al., 2021) or for graphs drawn from the same statistical distribution (Keriven et al., 2020; Gao et al., 2021).
Common among all these previous works are two themes limiting practical applicability: First and foremost, the class of ﬁlters to which results are applicable is often severely restricted.
The same is true for the class of considered graph shift operators; with non-normal operators (describing directed graphs) either explicitly or implicitly excluded.
Furthermore – when investigating transferability properties – results are almost exclusively available under the assumption that graphs are large and either discretize the same underlying ’continuous’ limit object sufﬁeciently well, or are drawn from the same statistical distributions.
While these are of course relevant regimes, they do not allow to draw conclusions beyond such asymptotic settings, and are for example unable to deal with certain spatial graphs, inapplicable to small-to-medium sized social networks and incapable of capturing the inherent multi-scale nature of molecular graphs (as further discussed below).
Finally, hardly any work has been done on relating the stability to input-signal perturbations to network properties such as the interplay of utilized ﬁlters or employed non-linearities.
The main focus of this work is to provide alleviation in this situation and develop a ’general theory of stability’ for GCNs – agnostic to the types of utilized ﬁlters, graph shift operators and non-linearities; with practically relevant transferability guarantees not contingent on potentially underlying limit objects.
To this end, Section 2 recapitulates the fundamentals of GCNs in a language adapted to our endeavour.
Sections 3 and 4 discuss stability to node- and edge-level perturbations.
Section 5 discusses stability to structural perturbations. Section 6 discusses feature aggregation and Section 7 provides numerical evidence.
GCN S VIA C OMPLEX A NALYSIS AND O PERATOR T HEORY
Throughout this work, we will use the label G to denote both a graph and its associated vertex set.
Taking a signal processing approach, we consider signals on graphs as opposed to graph embeddings:
Node-Signals: Node-signals on a graph are then functions from G to the complex numbers; i.e. elements of C | G | (with | G | the cardinality of G ).
We allow nodes i P G in a given graph to have weights µ i not necessarily equal to one and equip the space C | G | with an inner product according to x f, g y
“ ř i P G f p i q g p i q µ i to account for this.
We denote the hence created Hilbert space by (cid:96) 2 p G q .
Characteristic Operators: Fixing an indexing of the vertices, information about connectivity within the graph is encapsulated into the set of edge weights, collected into the adjacency matrix W and (diagonal) degree matrix D .
Together with the weight matrix M : “ diag ´ t µ i u | G | i “¯ , various standard geometry capturing characteristic operators – such as weighted adjacency matrix M ´ 1 W , graph Laplacian ∆ : “ M ´ 1 p D ´ W q and normalized graph Laplacian L : “ M ´ 1 D ´ 12 p D ´
W q D ´ 12 can then be constructed.
For undirected graphs, all of these operators are self-adjoint.
On directed graphs, they need not even be normal ( T ˚ T “ TT ˚ ).
We shall remain agnostic to the choice of characteristic operator; differentiating only between normal and general operators in our results.
Functional Calculus Filters:
A crucial component of GCNs are functional calculus ﬁlters, which arise from applying a function g to an underlying characteristic operator T ; creating a new operator g p T q .
Various methods of implementations exist, all of which agree if multiple are applicable:
G ENERIC F ILTERS :
If (and only if) T is normal, we may apply generic complex valued functions g to T :
Writing normalized eigenvalue-eigenvector pairs of T as p λ
i , φ i q | G | i “ 1 one deﬁnes g p T q ψ “ ř | G | i “ 1 g p λ
i
qx φ
i , ψ y (cid:96) 2 p G q φ i for any ψ P (cid:96) 2 p G q .
One has } g p T q} op “ sup λ P σ p T q | g p λ q| , with σ p T q denoting the spectrum of T .
If g is bounded, one may obtain the T -independent bound } g p T q} op ď } g } 8 .
Keeping in mind that g being deﬁned on all of σ p T q (as opposed to all of C ) is clearly sufﬁcient, we deﬁne a space of ﬁlters which will harmonize well with our concept of transferability discussed in
Section 5.
The introduced semi-norm will quantify the stability to perturbations in coming sections.
Deﬁnition 2.1.
Fix ω P C and C ą 0 .
Deﬁne the space F contω,C of continuous ﬁlters on C zt ω, ω u , to be the space of multilinear power-series’ g p
z
q
“ ř 8 µ,ν “ 0 a µν p ω ´
z q ´ µ p ω ´
z q ´ µ for which the semi-norm } g } F contω,C : “ řµ,ν ą 0 | µ ` ν | C µ ` ν ´ 1 | a µν | is ﬁnite.
Denoting by B (cid:15) p ω q Ď C the open ball of radius (cid:15) around ω , one can show that for arbitrary δ ą 0 and every continuous function g deﬁned on C zp B (cid:15)
p ω
q Y B (cid:15) p ω qq which is regular at inﬁnity – i.e. satisﬁes lim r Ñ`8 g p rz q “ c P C independent of which z ‰ 0 is chosen – there is a function f P F contω,C
so that | f p z q ´ g p z q| ď δ for all z P C zp B (cid:15) p ω q Y B (cid:15)
p ω qq .
In other words, functions in F contω,C can approximate a wide class of ﬁlters to arbitrary precision.
More details are presented in Appendix C.
E NTIRE F ILTERS
If T is not necessarily normal, one might still consistently apply entire (i.e. everywhere complex differentiable) functions to T .
Detail details on the mathematical background are given in Appendix D.
Here we simply note that such a function g is representable as an (everywhere convergent) power series g p
z q : “ ř 8 k “ 0 a gk z k
so that we may simply set g p T q “ ř 8 k “ 0 a gk ¨ T k .
For the norm of the derived operator one easily ﬁnds } g p T q} op ď
ř 8 k “ 0 | a gk |} T } kop using the triangle inequality.
While entire ﬁlters have the advantage that they are easily and efﬁciently implementable – making use only of matrix multiplication and addition – they suffer from the fact that it is impossible to give a } T } op -independent bound for } g p T q} op as for continuous ﬁlters.
This behaviour can be traced back to the fact that no non-constant bounded entire function exists (Bak & Newman, 2017).
H OLOMORPHIC F ILTERS :
To deﬁne functional calculus ﬁlters that are both applicable to nonnormal T and boundable somewhat more controlably in terms of T , one may relax the condition that g be entire to demanding that g be complex differentiable (i.e. holomorphic ) only on an open subset U Ď C of the complex plane.
Here we assume that U extends to inﬁnity in each direction (i.e. is the complement of a closed and bounded subset of C ).
For any g holomorphic on U and regular at inﬁnity we set (with p zId ´ T q ´ 1 the so called reolvent of T at z )
[Equation]
for any T whose spectrum σ p T q is completely contained in U .
Here we have used the notation g p8q “ lim r Ñ`8 g p rz q and taken D to an open set with nicely behaved boundary B D (more precisely a Cauchy domain; c.f. Appendix D).
We assume that D completely contains σ p T q and that its closure D is completely contained in U .
The orientation of the boundary B D is the usual positive orientation on D (such that D ’is on the left’ of B D ; cf.
Fig
Using elementary facts from complex analysis it can be shown that the resulting operator g p T q in (1) is independent of the speciﬁc choice of D (Gindler, 1966).
While we will present results below in terms of this general deﬁnition – remaining agnostic to numerical implementation methods for the most part – it is instructive to consider a speciﬁc exemplary setting with deﬁnite and simple numerical implementation of such ﬁlters: To this end, chose an arbitrary point ω P C and set U “ C zt ω u in the deﬁnitions above.
Any function g that is holomorphic on U and regular at 8 may then be represented by its Laurent series, which is of the form g p z q “ ř 8 k “ 0 b gk p
z ´ ω q ´ k (Bak & Newman, 2017).
For any T with σ p T q Ď U (i.e. ω R σ p T q ) evaluating the integral in (1) yields (c.f. Appendix D):
[Equation]
Such ﬁlters have already been employed successfully, e.g. in the guise of Cayley ﬁlters (Levie et al., 2019c), which are polynomials in z `
i z ´ i “ 1 `
i z
´ i .
We collect them into a designated ﬁlter space:
Deﬁnition 2.2.
For a function g p z q “ ř 8 k “ 0 b gk p
z
´
ω
q
´
k on U : “ C zt ω u deﬁne the semi-norm } g } F holω,C : “ řk “ 1 | b gk | kC k ´ 1 for C ą 0 .
Denote the set of such g for which } g } F holω,C ă 8 by F holω,C .
In order to derive } T } op -independent bounds for } g p T q} op , we will need to norm-bound the resolvents appearing in (1) and (2).
If T is normal, we simply have }p zId ´ T q ´ 1 } op “ 1 { dist p z, σ p T qq .
In the general setting, following Post (2012), we call any positive function γ T satisfying }p zId ´ T q ´ 1 } op ď γ T p z q on C z σ p T q a resolvent proﬁle of T .
Various methods (e.g. Szehr (2014); MichaelGil (2012)) to ﬁnd resolvent proﬁles.
Most notably Bandtlow (2004b) gives a resolvent proﬁle solely in terms of 1 { dist p z, σ p T qq and the departure from normality of T .
We then ﬁnd the following result:
Lemma 2.3.
For holomorphic g and generic T we have } g p T q} op ď | g p8q|` 12 π ű
B D | g p z q| γ T p z
q d | z | .
Furthermore we have for any T with γ T p ω q ď C , that } g p T q} op ď } g } F holω,C as long as g P F holω,C .
Lemma 2.3 (proved in Appendix E) ﬁnally bounds } g p T q} op independently of T , as long as appearing resolvents are suitably bounded; which – importantly – does not force } T } op to be bounded.
Non-Linearities & Connecting Operators:
To each layer of our GCN, we associate a (possibly) non-linear and L n -Lipschitz-continuous function ρ n : C Ñ C satisfying ρ n p 0 q “ 0 which acts point-wise on signals in (cid:96) 2 p G n q .
This deﬁnition allows to choose ρ n “ | ¨ | , ReLu , Id or any sigmoid function shifted to preserve zero.
To account for recently proposed networks where input- and ’processing’ graphs are decoupled (Alon & Yahav, 2021; Topping et al., 2021), and graph pooling layers (Lee et al., 2019), we also allow signal representations in the hidden network layers n to live in varying graph signal spaces (cid:96) 2 p G n q .
Connecting operators are then (not necessarily linear) operators
P n : (cid:96) 2 p G n ´ 1 q Ñ (cid:96) 2 p G n q connecting the signal utilized of subsequent layers.
We assume them to be
R n -Lipschitz-continuous ( } P n p f q ´ P n p g q} (cid:96) 2 p G n ´ 1 q ď R n } f ´ g } (cid:96) 2 p G n q q and triviality preserving ( P n p 0 q “ 0 ).
For our original node-signal space we also write (cid:96) 2 p G q ” (cid:96) 2 p G 0 q .
Graph Convolutional Networks: A GCN with N layers is then constructed as follows:
Let us denote the width of the network at layer n by K n .
The collection of hidden signals in this layer can then be thought of a single element of
[Equation]
Further let us write the collection of functional calculus ﬁlters utilized to generate the representation of this layer by t g nij p¨q : 1 ď j ď
K n ´ 1 ; 1 ď i ď K n u .
Further denoting the characteristic operator of this layer by T n , the update rule (c.f. also Fig. 2) from the representation in L n ´ 1 to L n is then deﬁned on each constituent in the direct sum L n as
[Equation]
We also denote the initial signal space by L in : “ L 0 and the ﬁnal one by L out : “ L N .
The hence constructed map from the initial to the ﬁnal space is denoted by Φ : L in Ñ L out .
S TABILITY TO I NPUT S IGNAL P ERTURBATIONS
In order to produce meaningful signal representations, a small input signal change should produce only a small variation in the output of our GCN.
This property is quantiﬁed by the Lipschitz constant of the map Φ associated to the network, which is estimated by our ﬁrst result below.
Theorem 3.1.
With the notation of Section 2 let Φ N : L in Ñ L out be the map associated to an
N -layer GCN.
We have with B n : “ [Equation] sup λ P σ p T n
[Equation] K n | g nij p λ q| 2 for all f, h P L in that
[Equation]
if T n is normal.
For general T n we have for all t g ij u entire, holomorphic and in F holω,C respectively:
[Equation]
Appendix F contains the corresponding proof and discusses how the derived bound are not necessarily tight for sparsely connected layers.
After Lipschitz constants of connecting operators and nonlinearities are ﬁxed, the stability constant of the network is completely controlled by the t B n u ; which for normal T n in turn are controlled by the interplay of the utilized ﬁlters on the spectrum of T n .
This allows to combine ﬁlters with sup λ P σ p T n q | g nij p λ q| “ O p 1 q but supported on complimentary parts of the spectrum of T n while still maintaining B n “ O p 1 q instead of O p
[Equation] ´ 1 q .
In practice one might thus penalize a ’multiple covering’ of the spectrum by more than one ﬁlter at a time during training in order to increase stability to input signal perturbations.
If T n is not normal but ﬁlters are holomorphic, an interplay persists – with ﬁlters now evaluated on a curve and at inﬁnity.
S TABILITY TO E DGE P ERTURBATIONS
Operators capturing graph-geometries might only be known approximately in real world tasks; e.g. if edge weights are only known to a certain level of precision.
Hence it is important that graph convolutional networks be insensitive to small changes in the characteristic operators t T n u .
Since we consider graphs with arbitrary vertex weights t µ g u g P G , we also have to consider the possibility that these weights are only known to a certain level of precision.
In this case, not only do the characteristic operators T n , r T n differ, but also the the spaces (cid:96) 2 p G q , (cid:96) 2 p r G q on which they act.
To capture this setting mathematically, we assume in this section that there is a linear operator J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q facilitating contact between signal spaces (of not-necessarily the same dimension).
We then measure closeness of characteristic operators in the respective spaces by considering the generalized normdifference }p JT ´ r TJ q} ; with J translating between the respective spaces.
Before investigating the stability of entire networks we ﬁrst comment on single-ﬁlter stability.
For normal operators we then ﬁnd the following result, proved in Appendix B building on ideas ﬁrst developed in (Wihler, 2009).
Lemma 4.1.
Denote by } ¨ } F the Frobenius norm and let T and r T be normal on (cid:96) 2 p G q and (cid:96) 2 p r G q respectively.
Let g be Lipschitz continuous with Lipschitz constant D g .
For any linear
J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q we have } g p r T q J ´
Jg p T q} F ď D g } r TJ ´ JT } F .
Unfortunately, scalar Lipschitz continuity only directly translates to operator functions if they are applied to normal operators and when using Frobenius norm (as opposed to e.g. spectral norm).
For general operators we have the following somewhat weaker result, proved in Appendix G:
Lemma 4.2.
Let T, r T be operators on on (cid:96) 2 p G q , (cid:96) 2 p r G q with } T } op , } r T } op ď C .
Let J : (cid:96) 2 p G q Ñ (cid:96) 2 p r
G q be linear.
With K g “ 12 π ű
B D 1 | z | γ T p z q γ r T p
z q| g p z q| d | z | for g holomorphic and
K g “ řk “ 1 | a gk | kC k ´ 1 for g entire, we have } g p T q J ´
Jg p r T q} op ď K g ¨ } JT ´ r TJ } op .
For a complete GCN we then ﬁnd the following result, proved in Appendix G:
Theorem 4.3.
Let Φ N , r Φ N be the maps associated to N -layer graph convolutional networks with the same non-linearities and ﬁlters, but based on different graph signal spaces (cid:96) 2 p G q , (cid:96) 2 p r G q , characteristic operators T n , r T n and connecting operators P n , r P n .
Assume B n , r B n ď B as well as R n , r R n ď R and L n ď L for some B, R, L ą 0 and all n ě 0 .
Assume that there are identiﬁcation operators
J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q ( 0 ď n ď N ) commuting with non-linearities and connecting operators in the sense of } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q “ 0 and }
ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q “ 0 .
Depending on whether normal or arbitrary characteristic operators are used, deﬁne D 2 n : “ ř j P K n ´
ř i P K n D 2 g nij or D 2 n : “ ř j P K n ´
ř
i P K n K 2 g nij .
Choose D such that D n ď D for all n .
Finally assume that } J n T n ´ r T n J n } ˚ ď δ and with ˚ “ F if both operators are normal and ˚ “ op otherwise.
Then we have for all f P L in and with J n the operator that the K n copies of J n induce through concatenation that } r Φ p J 0 f q ´ J N Φ p f q} Ă L out ď N ¨ DRL ¨ p BRL q N ´ 1 ¨ } f } L in ¨ δ .
The result persists with slightly altered constants, if identiﬁcation operators only almost commute with non-linearities and/or connecting operators, as Appendix H further elucidates.
Since we estimated various constants ( B n , D n , ... ) of the individual layers by global ones, the derived stability constant is clearly not tight.
However it portrays requirements for stability to edge level perturbations well:
While the (spectral) interplay of Section 3 remains important, it is now especially large single-ﬁlter stability constants in the sense of Lemmata 4.1 and 4.2 that should be penalized during training.
S TABILITY TO STRUCTURAL P ERTURBATIONS : T RANSFERABILITY
While the demand that } r TJ ´ JT } be small in some norm is well adapted to capture some notions of closeness of graphs and characteristic operators, it is too stringent to capture others.
As an illustrative example, further developed in Section 5.2 and numerically investigated in Section 7 below, suppose we are given a connected undirected graph with all edge weights of order
O p 1 { δ q .
With the
Laplacian as characteristic operator (governing heat-ﬂow in Physics (Cole, 2011)), we may think of this graph as modelling an array of coupled heat reservoirs with edge weights corresponding to heat-conductivities.
As 1 { δ Ñ 8 , the conductivities between respective nodes tend to inﬁnity, heat exchange is instantaneous and all nodes act as if they are fused together into a single large entity – with the graph together with its characteristic operator behaving as an effective one-dimensional system.
This ’convergent’ behaviour is however not reﬂected in our characteristic operator, the graph Laplacian
∆ δ : Clearly } ∆ δ } op “ 1 { δ ¨ } ∆ 1 } op Ñ 8 as 1 { δ Ñ 8 .
Moreover, we would also expect a Cauchylike behaviour from a ’convergent system’, in the sense that if we for example keep 1 { δ a ´ 1 { δ b “constant but let p 1 { δ a q , p 1 { δ b q Ñ 8 we would expect } ∆ δ a ´ ∆ δ b } op Ñ 0 by a triangle-inequality argument.
However, we clearly have } ∆ δ a ´ ∆ δ b } op “ | 1 { δ a ´ 1 { δ b |¨} ∆ 1 } op “ } ∆ 1 } op , which does not decay.
The situation is different however, when considering resolvents of the graph Laplacian.
An easy calculation (c.f. Appendix I) yields }p ωId ´ ∆ δ b q ´
´
p ωId ´ ∆ δ a q ´ 1 } op “ O p δ a ¨ δ b q so that we recover the expected Cauchy behaviour.
What is more, we also ﬁnd the convergence p ωId ´ ∆ δ q ´ 1 Ñ P
¨ p ω ´
q ´ 1 ; where P 0 denotes the projection onto the one-dimensional lowest lying eigenspace of the ∆ δ s (spanned by the vectors with constant entries).
We may interpret p ω ´
q ´as the resolvent of the graph Laplacian of a singleton (since such a Laplacian is identically zero) and thus now indeed ﬁnd our physical intuition about convergence to a one-dimensional system reﬂected in our formulae.
Motivated by this example, Section 5.1 develops a general theory for the difference in outputs of networks evaluated on graphs for which the resolvents R ω : “ p ωId ´ T q ´ 1 and r
R ω : “ p ωId ´ r T q ´ 1 of the respective characteristic operators are close in some sense.
Subsequently,
Section 5.2 then further develops our initial example while also considering an additional setting.
5. G ENERAL T HEORY
Throughout this section we ﬁx a complex number ω P C and for each operator T assume ω, ω R σ p T q .
This is always true for ω with | ω | ě } T } op , but if T is additionally self adjoint one could set ω “ i .
If T is non-negative one might choose ω “ p´ 1 q ).
As a ﬁrst step, we then note that the conclusion of
Lemma 4.1 can always be satisﬁed if we chose J ” 0 .
To exclude this case – where the application of J corresponds to losing too much information – we follow Post (2012) in making the following deﬁnition:
Deﬁnition 5.1.
Let J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q and r J : (cid:96)
2 p r G q Ñ (cid:96) 2 p G q be linear, and let T ( r T ) be operators on ( (cid:96) 2 p G q ) ( (cid:96) 2 p r G q ).
We say that J and r J are (cid:15) -quasi-unitary with respect to T , r T and ω if
[Equation]
The motivation to include the resolvents in the norm estimates (4) comes from the setting where
T “ ∆ is the graph Laplacian and ω “ p´ 1 q .
In that case, the left equation in (4 is for example automatically fulﬁlled when demanding }p Id ´ r JJ q f } 2 (cid:96) 2 p G q ď (cid:15) p} f } 2 ` E ∆ p f qq 12 , with E ∆ p¨q “ x¨ , ∆ ¨y (cid:96) 2 p G q the (positive) energy form induced by the Laplacian ∆ (Post, 2012).
This can thus be interpreted as a relaxation of the standard demand }p Id ´ r JJ q} op ď (cid:15) .
Relaxing the demands of
Section 4, we now demand closeness of resolvents instead of closeness of operators:
Deﬁnition 5.2.
If, for ω P C and linear J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q the resolvents R ω and r R ω satisfy }p r R ω
J ´ JR ω q f } (cid:96) 2 p r G q ď (cid:15) } f } (cid:96) 2 p G q for all f P (cid:96) 2 p G q , T and r T are called ω - (cid:15) -close with identiﬁcation operator J .
If additonally }p r R ˚ ω J ´ JR ˚ ω q f } (cid:96) 2 p r G q ď (cid:15) } f } (cid:96) 2 p G q , they are doubly ω - (cid:15) -close .
Our ﬁrst result establishes that operators being (doubly-) ω - (cid:15) -close indeed has useful consequences:
Lemma 5.3.
Let T ( r T ) be operators on (cid:96) 2 p G q ( (cid:96) 2 p r G q ).
If these operators are ω - (cid:15) -close with identiﬁcation operator J , and } R ω } op , } r R ω } op ď C we have } Jg p T q ´ g p r T q J } op ď K g ¨ }p r R ω J ´
JR ω q} op with K g “ 12 π ű
B D p 1 ` | z ´ ω | γ T p z qqp 1 ` | z ´ ω | γ r T p z
qq| g p
z q| d | z | for holomorphic g ,
K g “ } g } F holω,C if g P F holω,C and K g “ } g } F contω,C for T , r T normal and doubly ω - (cid:15) -close.
This result may then be extended to entire networks, as detailed in Theorem 5.4 below whose statement persists with slightly altered stability constants, if identiﬁcation operators only almost commute with non-linearities and/or connecting operators.
Proofs are contained in Appendix J.
Theorem 5.4.
Let Φ N , r Φ N be the maps associated to N -layer graph convolutional networks with the same non-linearities and functional calculus ﬁlters, but based on different graph signal spaces (cid:96) 2 p G n q , (cid:96) 2 p r G n q , characteristic operators T n , r T n and connecting operators P n , r P n .
Assume B n , r B n ď
B as well as R n , r R n ď R and L n ď L for some B, R, L ą 0 and all n ě 0 .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q ( 0 ď n ď N ) commuting with nonlinearities and connecting operators in the sense of } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q “ 0 and } ρ n p J n f q´
J n ρ n p f q} (cid:96) 2 p r G n q “ 0 . deﬁne D 2 n : “ ř j P K n ´
ř
i P K n K 2 g nij with K g nij as in Lemma 5.3.
Choose D such that D n ď D for all n .
Finally assume that } J n p ωId ´ T n q ´ 1 ´ p ωId ´ r T n q ´ 1 J n } op ď (cid:15) .
If ﬁlters in F contω,C are used, assume additionally that } J n pp ωId ´ T n q ´ 1 q ˚ ´pp ωId ´ r T n q ´ 1 q ˚ J n } op ď (cid:15) .
Then we have for all f P L in and with J n the operator that the K n copies of J n induce through concatenation that } r Φ N p J 0 f q ´ J N Φ N p f q} Ă L out ď N ¨ DRL ¨ p BRL q N ´ 1 ¨ } f } L in ¨ (cid:15) .
5. E XEMPLARY A PPLICATIONS
Collapsing Strong Edges: We ﬁrst pick our example from the beginning of section 5 up again and generalize it signiﬁcantly:
We now consider the graph that we collapse to a single node to be a sub-graph (of strong edges) embedded into a larger graph.
Apart from coupled heat reservoirs, this setting also e.g. captures the grouping of close knit communities within social networks into single entities, the scale-transition of changing the description of (the graph of) a molecule from individual atoms interacting via the coulomb potential Z 1 Z 2 { R (with R the distance
and Z 1 , Z 2 atomic charges) to the interaction of (functional) groups comprised of closely co-located atoms, or spatial networks if weights are set to e.g. inverse distances.
In what follows, we shall consider two graphs with vertex sets G and r G .
We consider G to be a subset of the vertex set r G and think of the graph corresponding to G as arising in a collapsing procedure from the ’larger’ graph r G .
More precisely, we assume that the vertex set r G can be split into three disjoint subsets r G “ r G Latin Ť
r G Greek Ť t‹u
(c.f.
also Fig. 3).
We assume that the adjacency matrix Ă W when restricted to Latin vertices or a Latin vertex and the exceptional node ’ ‹ ’ is of order unity p Ą W ab , Ă W a ‹ “ O p 1 q , @ a, b P r G Latin q .
For Greek indices, we assume that we may write Ă W αβ “ ω αβ δ and Ă W α ‹ “ ω α ‹ δ such that p ω αβ , ω α ‹ “ O p 1 q for all α, β P r G Greek .
We also assume that the sub-graph corresponding to vertices in r G Greek Ť t‹u is connected.
We then take G “ r G Latin Ť t‹u (c.f. again Fig. 3).
The adjacency matrix W on this graph is constructed by deﬁning W ab “ Ă W ab , @ a, b P r
G Latin and setting (with W a ‹ ” W ‹ a )
W ‹ a
[Equation]
Latin ¯ .
We also allow our graph r G to posses node-weights t r µ r g u r g P r G that are not necessarily equal to one.
The Laplace operator ∆ r G acting on the graph signal space (cid:96) 2 p r G q induces a positive semi-deﬁnite and convex energy form on this signal space via E r G p u q : “ x u, ∆ r G u y (cid:96) 2 p r G q “ ř g,h P r G Ă W gh | u p g q´ u p h q| 2 .
Using this energy form, we now deﬁne a set comprised of | G | signals, all of which live in (cid:96) 2 p r G q .
These signals are used to facilitate contact between the respective graph signal spaces (cid:96) 2 p G q and (cid:96) 2 p r G q .
Deﬁnition 5.5.
For each g P G , deﬁne the signal ψ δg P (cid:96)
2 p r G q as the unique solution to the convex optimization program
[Equation]
Given the boundary conditions, what is left to determine in the above optimization program are the ’Greek entries’ ψ δg p α q of each ψ δg .
As Appendix K further elucidates, these can be calculated explicitly and purely in terms of the inverse of ∆ r G restricted to Greek indices as well as (sub-)columns of the adjacency matrix Ă W .
Node-weights on G are then deﬁned as µ δg : “ ř h P r G ψ δg p h q ¨ r µ h .
We denote the corresponding signal space by (cid:96) 2 p G q .
Importantly, one has µ δa Ñ r µ a for any Latin index and µ δ ‹ Ñ r µ ‹ ` ř α P r G Greek r µ α as δ Ñ 0 ; which recovers our physical intuition about heat reservoirs.
To translate signals from (cid:96) 2 p G q to (cid:96) 2 p r G q and back, we deﬁne two identiﬁcation operators J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q and r J : (cid:96)
2 p r G q Ñ (cid:96) 2 p G q via Jf : “ ř g P G f p g q ¨ ψ δg and p r Ju qp g q : “ x u, ψ δg y (cid:96) 2 p r
G q { µ δg for all f P (cid:96) 2 p G q , u P (cid:96) 2 p r G q
and g P G .
Our main theorem then states the following:
Theorem 5.6.
With deﬁnitions and notation as above, there are constants K 1 , K 2 ě 0 such that the operators J and r J are p K 1 ?
δ q -quasi-unitary with respect to ∆ r G , ∆ G and ω “ p´ 1 q .
Furthermore, the operators ∆ r G and ∆ G are p´ 1 q - p K 2 ?
δ q close.
with identiﬁcation operator J .
Appendix K presents the (fairly involved) proof of this result.
Importantly, the size of the constants
K 1 , K 2 is independent of the cardinality (or more precisely the total weight) of r G Latin , implying that
Theorem 5.6 also remains applicable in the realm of large graphs.
Finally we note, that this stability result is contingent on the use of the (un-normalized) graph Laplacian (c.f. Appendix L):
Theorem 5.7.
In the setting of Theorem 5.6 denote by T ( r T ) adjacency matrices or normalized graph Laplacians on (cid:96) 2 p G q ( (cid:96) 2 p G q ).
There are no functions η 1 , η 2 : r 0 , 1 s Ñ R ě 0 with η
i p
δ q Ñas δ Ñ
( i “ 1 , 2 ), families of identiﬁcation operators J δ , r J δ and ω P C so that J δ and r J δ are η 1 p δ q -quasi-unitary with respect to r T , T and ω while the operators r T and T remain ω - η 2 p
δ q close.
The Realm of Large Graphs: In order to relate our transferability framework to the literature, we consider an ’increasing’ sequence of graphs ( G n Ď G n ` 1 ) approximating a limit object, so that the transferability framework of Levie et al.
(2019a) is also applicable.
We choose the limit object to be the circle of circumference 2 π and our approximating graphs to be the closed path-graph on N vertices equidistantly embedded into the circle (c.f. Fig 4).
With h “ 2 π { N the node-distance, we set weights to 1 { h 2 ; ensuring consistency with the ’continuous’ Laplacian in the limit N Ñ 8 .
More details are presented in Appendix M, which also contains the proof of the corresponding transferability result:
Theorem 5.8.
In the above setting choose all node-weights equal to one and N to be odd for deﬁniteness.
There exists constants K 1 , K 2 “ O p 1 q so that for each N ě 1 , there exist identiﬁcation operators J, r J mapping between (cid:96) 2 p G N q and (cid:96) 2 p G N ` 1 q so that J and r J are p K 1 { N q -quasi-unitary with respect to ∆ G N , ∆ G N ` 1 and ω “ p´ 1 q .
Furthermore, the operators ∆ G N and ∆ G N ` 1 are p´ 1 q - p K 2 { N q close with identiﬁcation operator J .
Lemma 5.3 then implies an O p 1 N q -decay of } g p T q J ´
Jg p r T q} op for ﬁxed g .
This reduces to an
O p ?
NN q -decay for Levie et al. (2019a) (ibid.
Theorem 5 , pt. 3 ) assuming a similar decay of operatordistances.
Our framework might this capture transferability properties other approaches could miss.
G RAPH L EVEL S TABILITY
To solve tasks such as graph classiﬁcation or regression over multiple graphs, graphs of varying sizes need to be represented in a common feature space.
Here we show that aggregating node-level features into such graph level features via p -norms ( } f } (cid:96) p p G q : “ p ř g P G | f g | p µ g q 1 { p ) preserves stability.
To this end, let L out be a target space of a GCN in the sense of (3).
On each of the (in total K out ) (cid:96) 2 p G out q summands of L out , we may apply the map f i ÞÑ } f i } (cid:96) p p G out q .
Stacking these maps, we build a map from L out to R K out .
Concatenating the map Φ N associated to an N -layer GCN with this map yields a map from L in to R K out .
We denote it by Ψ pN and ﬁnd:
Theorem 6.1.
For p ě 2 we have in the setting of Theorem 3.1 that } Ψ pN p f q ´ Ψ pN p h q} R K out ď ´ś
Nn “ 1 L n R n B n ¯ ¨ } f ´ h } L in .
In the setting of Theorem 4.3 or 5.4 and under the additional assumption that the ’ﬁnal’ identiﬁcation operator J N satisﬁes ˇˇ } J N f i } (cid:96) k p r G N q ´ } f i } (cid:96) k p G N q ˇˇ ď δ
¨
K
¨ } f i } (cid:96) 2 p
G N q for all f i P (cid:96) 2 p G N q , we have } Ψ pN
p
f
q ´ r Ψ pN p J 0 f q} R K out ď p N ¨ DRL ` K ¨ p BRL qq ¨ p BRL q N ´ 1 ¨ } f } L in ¨ δ .
Derived stability results thus persist (under mild assumptions) if graph level features are aggregated via p -norms.
Appendix N contains the corresponding proof.
N UMERICAL R ESULTS
We focus on investigating structural perturbations, as corresponding results are most involved and novel:
We ﬁrst consider a graph on 5 nodes with an adjacency matrix A with O p 1 q -entries (c.f. 30 in Appendix O).
We then scale A by 1 { δ a and 1 { δ b (with 1 δ a ´ 1 δ b “ 1 ) respectively and consider the norm-difference between associated Laplacians and resolvents.
Fig.
6 (a) then illustrate the theoretical result (c.f. Section 5) that resolvent- instead of Laplacian-differences capture the convergence behaviour.
Embedding the considered graph into a larger graph ( Ă W P R 8 ˆ 8 ; c.f. (31) in Appendix O), we consider the collapsing edge setting of Section 5.2 in Fig.
6 (b).
As expected, the corresponding resolvents do approach each other as δ Ñ 0 .
Contrary to the theoretical bound in Lemma 5.3, differences of resolvent-monomials decrease as their power k increases.
Beyond small graphs – inaccessible to traditional asymptotic methods – our method is also applicable to the large-graph setting:
Fig.
7 picks up the example of an ’increasing’ graph sequence ’approximating’ the circle again.
As predicted in Section 5.2, the difference in resolvents decays ( 9 1 N ).
Fig.
10 in Appendix O shows how the difference in Laplacians diverges instead.
Hence our framework might capture stability properties traditional approaches could miss.
Finally, we investigate the transferability of a two-layer GCN with 16 nodes per hidden Layer combined with the aggregation method of Section 6 into a graph-level map Ψ p 2 .
Filters are of the form (2) up to order k “ 11 .
Coefﬁcients t b gk u are sampled uniformly from r´ 100 , 100 s .
Feature vectors are generated on the QM 7 dataset.
There each graph represents a molecule; nodes correspond to individual atoms.
The adjacency matrix is given by Ă
W ij “ p Z
i
Z
j q{| x i ´ x j | with Z i ( x i ) the atomic charge (equilibrium position) of atom i .
We choose node-weights as r µ i “ Z i and the Laplacian as characteristic operator.
Leading up to Fig.
8 we consider the graph of methane ( 5 Nodes; one Carbon ( Z 1 “ 6 ); 4 Hydrogen nodes ( Z i ą 1 “ 1 )) and deﬂect one of the Hydrogen atoms ( i “ 2 ) out of equilibrium closer to the Carbon atom.
Neglecting changes in inter-Hydrogen distances, we consider the transferability of the entire GCN between this graph and an effective description combining Carbon and deﬂected Hydrogen into a single node ‹ ( µ ‹ “ 7 ).
With J translating from effective to original description, we consider } Ψ p 2 p f q ´ Ψ p 2 p
Jf q} R 16 (averaged over 100 random unit-norm choices of f ) as a function of | x 1 ´ x 2 | ´ 1 .
At equilibrium (corresponding to the left-end of
Fig.
8) the transferability error is O p 1 q but decreases fast with decreasing Carbon-Hydrogen distance; with the choice of representation (effective vs. original) quickly becoming insigniﬁcant for generated feature
D ISCUSSION
We developed a well founded framework capturing stability properties of general graph convolutional networks.
We related node-level stability to to spectral covering properties and edge-level stability to
Lipschitz constants of utilized ﬁlters.
For non-normal characteristic operators, tools from complex analysis provided grounds for derived stability properties.
We introduced a new notion of stability to structural perturbations, highlighted the importance of the resolvent and detailed how the developed line of thought captures relevant settings of structural changes such as the collapse of a strongly connected sub-graph to a node.
There – precisely if the graph Laplacian is employed – the transferability error can be bounded in terms of the inverse characteristic coupling strength on the sub-graph.
A N OTATIONAL C ONVENTIONS B S OME C ONCEPTS IN L INEAR A LGEBRA
In the interest of self-containedness we provide a brief review of some concepts from linear algebra utilized in this work that might potentially be considered more advanced.
Presented results are all standard; a very thorough reference is Michael Reed (1981)
Hilbert Spaces: To us, a Hilbert space — often denoted by H — is a vector space over the complex numbers which also has an inner product — often denoted by x¨ , ¨y H .
Prototypical examples are given by the Euclidean spaces C d with inner product x x, y y C d : “ ř di “ 1 x i y i .
Associated to an inner product is a norm, denoted by } ¨ } H and deﬁned by } x } H : “
[Equation]
x x, x y H for x P H .
Direct Sums of Spaces: Given two potentially different Hilbert spaces H and p H , one can form their direct sum H ‘ p H .
Elements of H ‘ p H are vectors of the form p a, b q , with a P H and b P p H .
Addition and scalar multiplication are deﬁned in the obvious way by
[Equation]
for a, c P H , b, d P p H and λ P C .
The inner product on the direct sum is deﬁned by
[Equation]
As is readily checked, this implies that the norm } ¨ } H ‘ p H on the direct sum is given by
[Equation]
Standard examples of direct sums are again the Euclidean spaces, where one has C d “ C n ‘ C m if m ` n “ d , as is easily checked.
One might also consider direct sums with more than two summands, writing C d “ ‘ di “ 1 C for example.
In fact, one might also consider inﬁnite sums of Hilbert spaces:
The space ‘ 8 i
[Equation]
1 H i is made up of those elements a “ p a 1 , a 2 , a 3 , ...
q with a i P H i for which the norm
[Equation]
is ﬁnite.
This means for example that the vector p 1 , 0 , 0 , 0 , ...
q is in ‘ 8 i “ 1 C , while p 1 , 1 , 1 , 1 , ...
q is not.
Direct Sums of Maps: Suppose we have two collections of Hilbert spaces t H
i u Γ i “ 1 , t r H
i u Γ i “ 1 with Γ P N or Γ “ 8 .
Suppose further that for each i ď Γ (resp.
i ă Γ ) we have a (not necessarily linear) map J i : H
i Ñ r H i .
Then the collection t J i u Γ i “ 1 of these ’component’ maps induce a ’composite’ map
[Equation]
between the direct sums.
Its value on an element a “ p a 1 , a 2 , a 3 , ...
q P ‘ Γ i “ 1 H i is deﬁned by
[Equation]
Strictly speaking, one has to be a bit more careful in the case where Γ “ 8 to ensure that } J p a q} ‘ 8 i “ 1 r H i ‰ 8 .
This can however be ensured if we have } J i p
a i q} r H i ď C } a i } H i for all 1 ď i and some C independent of all i , since then } J p a q} ‘ 8 i “ 1 r H i ď C } a } ‘ 8 i “ 1 H i ď 8 .
If each J i is a linear operator, such a C exists precisely if the operator norms (deﬁned below) of all J i are smaller than some constant.
Operator Norm: Let J : H Ñ r H be a linear operator between Hilbert spaces.
We measure its ’size’ by what is called the operator norm, denoted by } ¨ } op and deﬁned by
[Equation]
Adjoint Operators Let J : H Ñ r H be a linear operator from the Hilbert space H to the Hilbert space r H .
Its adjoint J ˚ : r H Ñ H is an operator mapping in the opposite direction.
It is uniquely determined by demanding that
[Equation]
holds true for arbitrary f P H and u P r H .
Normal Operators:
If a linear operator ∆ : H Ñ H maps from and to the same Hilbert space, we can compare it directly with its adjoint.
If ∆∆ ˚ “ ∆ ˚ ∆ , we say that the operator ∆ is normal.
Special instances of normal operators are self-adjoint operators, for which we have the stronger property ∆ “ ∆ ˚ .
If an operator is normal, there are unitary maps U : H Ñ H diagonalizing ∆ as
[Equation]
[Equation] [Equation]
n
[Equation]
, with eigenvalues in C .
We call the collection of eigenvalues the spectrum σ p ∆ q of ∆ .
If dim H “ d , we may write σ p ∆ q “ t λ u di “ 1 .
It is a standard exercise to verify that each eigenvalue satisﬁes | λ i | ď } ∆ } op .
Associated to each eigenvalue is an eigenvector φ i .
The collection of all (normalized) eigenvectors forms an orthonormal basis of H .
We may then write
[Equation]
Resolvent of an Operator:
Given an operator T on some Hilbert space H , we have by deﬁnition that the operator p T ´
z q : H Ñ H is invertible precisely if z ‰ σ p T q .
In this case we write
R z p T q “ p zId ´ T q ´and call this operator the resolvent of T at z .
If T is normal it can be proved that the norm of the resolvent satisﬁes
[Equation]
where dist p z, σ p ∆ qq denotes the minimal distance between z and any eigenvalue of ∆ .
For nonnormal operators, one can prove
[Equation]
with
[Equation]
as is proved in Bandtlow (2004a).
Frobenius Norm: Given two ﬁnite dimensional Hilbert spaces H 1 and H 2 with orthonormal bases t
φ 1 i u d 1 i “ 1 and t φ 1 i u d 1 i “ 1 , the Frobenius norm } ¨ } F of an operator A : H 1 Ñ H 2 may be deﬁned as
[Equation]
i
[Equation] [Equation]
ij |
[Equation]
with A ij the matrix representation of A with respect to the bases t φ 1 i u d 1 i “ 1 and t φ 1 i u d 1 i “ 1 .
It is a standard exercise to verify that this norm is indeed independent of any choice of basis and hence invariant under multiplying A with a unitary on either the left or the right side.
More precisely, if
U : H 2 Ñ H 2 and V : H 1 Ñ H 1 are unitary, we have
[Equation]
Frobenius norms can be used to transfer Lipschitz continuity properties of complex functions to the setting of functions applied to normal operators:
Lemma B.1.
Let g : C Ñ C be Lipschitz continuous with Lipschitz constant D g .
This implies
[Equation]
for normal operators X on H 2 , Y on H 1 and any linear map J : H 1 Ñ H 2 .
Proof.
This proof is a modiﬁed version of the proof in Wihler (2009).
Let U, W be unitary (with respect to the inner product x¨ ,
¨y H ) operators diagonalizing the normal operators X and Y as
[Equation]
Since the Frobenius norm is invariant under unitary transformations we ﬁnd
[Equation]
C A PPROXIMATING BOUNDED CONTINUOUS FILTERS
Let us recall Deﬁnition 2.1:
Deﬁnition C.1.
Fix ω P C and C ą 0 .
Deﬁne the space F contω,C of continuous ﬁlters on C zt ω, ω u , to be the space of multilinear power-series’ g p
z
q
“ ř 8 µ,ν “ 0 a µν p ω ´
z q ´ µ p ω ´
z q ´ µ for which the norm } g } F contω,C : “
[Equation] [Equation]
|
[Equation]
|
[Equation]
` ν | a µν | is ﬁnite.
We now prove that upon denoting by B (cid:15) p ω q Ď C the open ball of radius (cid:15) around ω , one can show that for arbitrary δ ą 0 and every continuous function g deﬁned on C zp B (cid:15) p ω
q Y B (cid:15) p ω qq which is regular at inﬁnity – i.e. satisﬁes lim r Ñ`8 g p rz q “ c P C independent of which z ‰ 0 is chosen – there is a function f P F contω,C
so that | f p z q ´ g p z q| ď δ for all z P C zp B (cid:15) p ω q Y B (cid:15)
p ω qq .
Making use of the Stone-Weierstrass theorem for complex functions, it sufﬁces to prove that for every point z in C zp B (cid:15) p ω
q Y B (cid:15) p ω
qq there are functions f and g in F contω,C for which
[Equation]
p z q
[Equation]
g
[Equation]
But this is obvious since p ω
´
z q ´ 1 is injective on C zp B (cid:15)
p ω
q Y B (cid:15) p ω qq .
D C OMPLEX A NALYSIS
A general reference for topics discussed in this section is Bak & Newman (2017).
For a complex valued function f of a single complex variable, the derivative of f at a point z 0 P C in its domain of deﬁnition is deﬁned as the limit
[Equation]
For this limit to exist, it needs to be independent of the ’direction’ in which z approaches z 0 , which is a stronger requirement than being real-differentiable.
A function is called holomorphic on an open set
U if it is complex differentiable at every point in U .
It is called entire if it is complex differentiable at every point in C .
Every entire function has an everywhere convergent power series representation
[Equation]
If a function g is analytic (i.e. can be expanded into a power series), we have
[Equation]
for any circle S Ď C encircling λ by Cauchy’s integral formula.
In fact, the integration contour need not be a circle S , but may be the boundary of any so called
Cauchy domain containing λ :
Deﬁnition D.1.
A subset D of the complex plane C is called a Cauchy domain if D is open, has a ﬁnite number of components (the closure of two of which are disjoint) and the boundary of B D of D is composed of a ﬁnite number of closed rectiﬁable Jordan curves, no two of which intersect.
Equation (7) forms the backbone of complex analysis.
Since the integral
[Equation]
is well deﬁned for holomorphic g p¨q and any operator T for which σ p T q and B D are disjoint (c.f.
e.g.
Post (2012) for details), we can essentially take (8) as a deﬁning equation through which one might apply holomorphic functions to operators.
While functions that are everywhere complex differentiable have a series representation according to (6), complex functions that are holomorphic only on
C zt ω
u have a series representation (called
Laurent series) according to
[Equation]
If these functions are assumed to be regular at inﬁnity, no terms with positive exponent are permitted and (changing the indexing)
we may thus write
[Equation]
Motivated by this, we now prove the following consistency result:
Lemma D.2.
With the notation of Section 2 we have for any k ě 1 and ω R σ p T q that
[Equation]
where we interpret the left hand side of the equation in terms of inversion and matrix powers.
Proof.
We ﬁrst note that we may write
[Equation]
for | λ ´ ω | ď } R ω p T q} using standard results in matrix analysis (namely the ’Neumann Characterisation of the Resolvent’ which is obtained by repeated application of a resolvent identity; c.f. Post (2012) for more details).
We thus ﬁnd
[Equation]
Using the fact that
[Equation]
then yields the claim.
E P ROOF OF L EMMA 2.
We want to prove the following:
Lemma E.1.
For holomorphic g and generic T we have } g p T q} op ď | g p8q| ` 12 π ű
B D | g p z q| γ T p z
q d | z | .
Furthermore we have for any T with γ T p ω q ď C , that } g p T q} op ď } g } F holω,C as long as g P F C,ω .
Proof.
We ﬁrst note
[Equation] [Equation]
The ﬁrst claim thus follows together with } R z p T q} op ď γ T p z q .
The second claim can be derived as follows:
[Equation]
F P ROOF OF T HEOREM 3.1 AND TIGHTNESS OF RESULTS
We want to prove the following:
Theorem F.1.
With the notation of Section 2 let Φ N : L in Ñ L out be the map associated to an
N -layer GCN.
We have
[Equation]
L n R n B n ¸ ¨ } f ´ h } L in with B n : “ b sup λ P σ p T n q ř j P K n
[Equation]
ř i P K n | g nij p λ q| 2 if T n is normal.
For general T n we have for all t g ij u entire, holomorphic and in F ω,C respectively:
[Equation]
Proof.
Given input signals f, h n P L in , let us – sticking to the notation introduced in Section 2 – denote the intermediate signal representations in the intermediate layers L n by f n , h n P L n .
With the update rule described in Section 2 and the norm induced on each L n as described in Appendix B, we then have
[Equation]
[Table]
We next note
[Equation]
where the second to last step is an application of the Cauchy Schwarz inequality.
Proceeding inductively and using our previously established estimates, this proves the claim for all settings in which T n is nor normal (using an additional application of the triangle inequality for the case of holomorphic ﬁlters).
To prove the claim for normal T n as well, we note that in this setting we have (writing p φ α , λ α q | G | α “for a normalozed eigenvalue-eigenvector sequence of T n ` 1 ) that we have
[Equation]
q
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
Here we applied Cauchy Schwarz once more in the second to last step and bounded
[Equation]
To see that these bounds are not necessarily tight, we may simply note that if we have a simple one-layer Network as depicted in Fig.
9 below, the stability can be tightened to
[Equation]
L
[Equation]
with with B n : “ max i “ a,b p sup λ P σ p T q | g i p λ
q|q as opposed to with B n : “ b sup λ P σ p T q ř i “ a,b |
g i p λ q| 2 if
T is normal; as an easy calculation shows.
G P ROOF OF L
EMMA 4.
We want to prove the following:
Lemma G.1.
Let T, r T be operators on on (cid:96) 2 p G q , (cid:96) 2 p r G q with } T } op , } r T } op ď C .
Let
J : (cid:96) 2 p G q Ñ (cid:96) 2 p r G q be arbitrary but linear.
With K g “ ř 8 k “ 1 | a gk | kC k ´ 1 for g entire and
K g “ 12 π ű
B D 1 z γ T p z
q γ r T p z q| g p z q| d | z | for g holomorphic, we have
[Equation]
Proof.
Let us ﬁrst verify the claim for entire g .
We ﬁrst note that
[Equation]
Thus, with } T } op , } r T } op ď C we ﬁnd
[Equation]
The claim now follows from applying the triangle inequality.
Now let us prove the bound for holomorphic g .
We ﬁrst note the following:
[Equation]
´ z ´
[Equation]
˙ .
Thus we have
[Equation]
H P ROOF OF T HEOREM 4.
We prove the following generalization of Theorem 4.3:
Theorem H.1.
Let Φ N , r Φ N be the maps associated to N -layer graph convolutional networks with the same non-linearities and functional calculus ﬁlters, but based on different graph signal spaces (cid:96) 2 p G q , (cid:96) 2 p r G q , characteristic operators T n , r T n and connecting operators P n , r P n .
Assume B n , r B n ď B as well as R n , r R n ď R and L n ď L for some B, R, L ą 0 and all n ě 0 .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q ( 0 ď n ď N ) almost commuting with nonlinearities and connecting operators in the sense of } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q ď δ 2 } f } (cid:96) 2 p G n q and } ρ n p J n f q´ J n ρ n p f q} (cid:96)
p r G n q ď δ 1 } f } (cid:96) 2 p G n q .
Depending on whether normal or arbitrary characteristic operators are used, deﬁne D 2 n : “ ř j P K n ´
ř i P K n D 2 g nij or D 2 n : “ ř j P K n ´
ř
i P K n K 2 g nij .
Choose
D such that D n ď D for all n .
Finally assume that } J n T n ´ r T n J n } ˚ ď δ and with ˚ “ F if both operators are normal and ˚ “ op otherwise.
Then we have for all f P L in and with J N the operator that the K N copies of J N induced through concatenation that
[Equation]
Proof.
For simplicity in notation
[Equation]
let us denote the hidden representation of J 0 f in Ă L n by r f n .
We then note the following
[Equation]
[Table]
[Equation]
We can bound the ﬁrst term by δ 1 B ¨ R ¨ p BRL q n ¨ } f } L in .
For the second term we ﬁnd
L
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Arguing as in the proof of 3.1 we can bound the ﬁrst term by LD ¨ δR ¨ p BRL q n } f } L in .
For the second term we ﬁnd,
[Equation]
arguing as above.
Iterating from n “ N to n “ 0 then yields the claim.
I T RANSFERABILITY : G ENERAL C ONSIDERATIONS
We ﬁrst prove the statement made at the beginning of Section 5 that
[Equation]
To this end denote the increasing sequence of eigenvalues (counted without multiplicity) of ∆ 1 by t
λ i u Mi “ 0 .
Recall that λ 0 “
Denote the sequence of projections on the corresponding eigenspaces by t P i u Mi “ 0 .
We have for the resolvent that
[Equation]
Thus we have for δ a , δ b small enough that ››››
[Equation]
Next we note the convergence p ωId ´ ∆ δ q ´ 1 Ñ P 0 ¨p ω ´
q ´ 1 .
But this is obvious, since for λ
i ‰we have
[Equation]
as δ Ñ 0 .
J P ROOFS OF L EMMA 5.3 AND T HEOREM 5.
Lemma J.1.
Let T and r T be characteristic operators on (cid:96) 2 p G q and (cid:96) 2 p r G q be respectively.
If these operators are ω - δ -close with identiﬁcation operator J , and } R ω } op , R ω } op ď C we have
[Equation]
op with K g “ ű
B D p 1 ` | z ´ ω | γ T p z qqp 1 ` | z ´ ω | γ r T p z
qq| g p
z q| d | z | if g is holomorphic and K g “ } g } F holω,C if g P F holω,C .
If T and r T are normal as well as doubly ω - δ -close
and g P F contω,C , we have
K g “ } g } F contω,C .
Proof.
We ﬁrst deal with the statement concerning holomorphic g .
To this end we note that Lemma 4.5.9 of Post (2012) proves
[Equation]
The claim then follows from
[Equation]
For g P F holω,C the claim is proved exactly as in the proof of Lemma 2.3.
For g P F contω,C we note that
[Equation]
Together with the result
[Equation]
established in the proof of Lemma 4.2, the claim then follows from the triangle inequality together with the deﬁnition of the semi-norm } g } F contω,C .
As in the previous section, we state a slightly more general version of our main theorem of this section:
Theorem J.2.
Let Φ , r
Φ be the maps associated to N -layer graph convolutional networks with the same non-linearities and functional calculus ﬁlters, but based on different graph signal spaces (cid:96) 2 p G n q , (cid:96) 2 p r G n q , characteristic operators T n , r T n and connecting operators P n , r P n .
Assume B n , r B n ď
B as well as R n , r R n ď R and L n ď L for some B, R, L ą 0 and all n ě 0 .
Assume that there are identiﬁcation operators J n : (cid:96) 2 p G n q Ñ (cid:96) 2 p r G n q ( 0 ď n ď N ) almost commuting with nonlinearities and connecting operators in the sense of } r P n J n ´ 1 f ´ J n P n f } (cid:96) 2 p r G n q ď δ 2 } f } (cid:96) 2 p G n q and } ρ n p J n f q ´ J n ρ n p f q} (cid:96) 2 p r G n q δ 1 } f } (cid:96) 2 p G n q . deﬁne D 2 n : “ ř j P K n ´
ř
i P K n K 2 g nij with K g n ij as in
Lemma 5.3.
Choose D such that D n ď D for all n .
Finally assume that } J n p ωId ´ T n q ´ 1 ´ p ωId ´ r
T n q ´ 1 J n } op ď δ .
If ﬁlters in F contω,C are used, assume additionally that } J n pp ωId ´ T n q ´ 1 q ˚ ´ pp ωId ´ r T n q ´ 1 q ˚ J n } op ď δ .
Then we have for all f P L in and with J N the operator that the K N copies of J N induced through concatenation that
[Equation]
Proof.
The proof proceeds in complete analogy to the one of Theorem 4.3.
K C OLLAPSING STRONG E DGES : P ROOFS AND FURTHER D ETAILS
We utilize the notation introduced in Section 5.2.
Beyond this, we denote the positive semi-deﬁnite form induced by the energy functional E r G by
[Equation]
We further use the notation E r G p u q : “ E r G p u, u q .
With
[Equation]
Similar considerations apply when r G is replaced by G .
Let us next solve the convex optimization program (5) introduced in Deﬁnition 5.5, restated here for convenience:
Deﬁnition K.1.
For each g P G , deﬁne the signal ψ δg P (cid:96)
2 p r G q as the unique solution to the convex optimization program
[Equation]
As a ﬁrst step we note that all entries of ψ g are real and non-negative, which follows since each summand in (9) is non-increasing under the map u ÞÑ | u | due to the reverse triangle
||
a |´| b || ď | a ´ b | .
To ﬁnd the explicit form of ψ g , ﬁx g P r G Latin Ť t‹u and denote by χ g P (cid:96) 2 p r G q the signal deﬁned by setting it to χ η p
h q “ δ hg for h P r G Latin Ť t‹u and η g p α q “ η αg with t η αg u α P r G Greek a set of | r G Greek | free parameters in R ď 0 .
We then have
[Equation]
By deﬁnition
[Equation]
χ g depends smoothly on the parameters t η αg u α P r G Greek .
Finding the minimizer of the convex optimization program (5) is then equivalent to ﬁnding the values t η αg u α P r G Greek at which we have
[Equation]
We note
[Equation]
Collecting these equations for all parameters into a matrix equation, we ﬁnd that the ’Greek entries’ of the vector ψ g are given explicitly by
[Equation]
with degrees in r G denoted by r d α .
Let us denote the restriction of ψ δg to Greek entries, thought of as a vector in C | r G Greek | by (cid:126)η δg .
Given the degree r d α corresponding to a Greek index, we decompose it as
[Equation]
with r d rα accounting for edges from α to other greek vertices
[Equation]
and V α accounting for edges from α to Latin vertices
[Equation]
Recall that we also may write
[Equation]
We may then write ¨
[Equation]
where we made the obvious deﬁnitions for the matrices L and V and denoted by (cid:126)ω ‹ the vector with entries ω α ‹ .
Let us also use the notation
[Equation]
Next we want to establish that h is invertible.
For this we ﬁrst note that that L is the graph Laplacian of the subgraph r G Greek ; which we assume to be connected.
Hence L is positive semi-deﬁnite with the eigenspace corresponding to the eigenvalue zero being spanned by (entry-wise) constant vectors.
Since all entries of ω ‹ are non-negative, the operator h is also positive semi-deﬁnite.
Since we assume that the vertex ‹ is connected to at least one other vertex in r G Greek , there is at least one entry in (cid:126)ω ‹ that is strictly greater than zero.
We show that this already implies that h is in fact also positive deﬁnite and hence invertible.
Indeed, for any (cid:126)v P C | r G Greek | we have
[Equation]
Both terms on the right hand side are non-negative.
If (cid:126)v is a constant (non-zero) vector, the ﬁrst term vanishes, but since at least one entry of ω ‹ is strictly positive, with all others being non-negative, the second term on the right hand side is strictly positive.
If (cid:126)v is non-constant, the ﬁrst term on the right hand side is larger than zero.
Hence h is positive deﬁnite and thus invertible.
Similarly one proves that (for any δ ě 0 ) the operator h ` δV is positive deﬁnite and hence invertible.
Thus we now know that the operator
[Equation]
utilized in (10) is indeed invertible.
We note (again with the restriction of ψ δg to Greek entries thought of as a vector in C | r G Greek | denoted by (cid:126)η δg ) that we may equivalently write (10) as
[Equation]
and
[Equation] [Equation]
‹‚ thought of as an element of C | r G Greek | .
To proceed, we now ﬁrst focus on the case g “ ‹ , for which we may write (11) equivalently as
[Equation]
Since (cid:126)ω ‹ is independent of δ , we may take the limit δ Ñ 0 and arrive at
[Equation]
which is uniquely solved by (cid:126)η 0 ‹ “ p 1 , 1 , 1 , .... q ” 1 Greek .
Since we assume δ ! 1 , we can now investigate the solution (cid:126)η δg for non-zero δ through perturbation theory.
We write
[Equation]
with (cid:126)ζ 0 ‹ “ 0 and ﬁnd from (12) – using h ¨ 1 Greek “ (cid:126)η δ ‹ – the deﬁning equation
[Equation]
From this we obtain the estimate
[Equation]
where we denote by (cid:96) 2 p r G Greek q the space graph signal space C | r G Greek | equipped with node weights t r µ g u g P r G Greek .
We note that both h and V are positive semi-deﬁnite and we thus obtain
[Equation]
for the minimal eigenvalues of the respective operators.
Hence
[Equation]
and thus also
[Equation]
Since } h ´ 1 } op “ 1 { λ min p h q we may write
[Equation]
From (11) we know that for g ‰ ‹ we have (cid:126)η δg “ 0 .
We now also want to bound } (cid:126)η δg } (cid:96)
2 p r G Greek q in terms of δ .
We will do this by establishing the relationship
[Equation] [Equation]
and then utilizing our estimate on } (cid:126)ζ δ ‹ } (cid:96) 2 p r G Greek q established above.
To prove (15), we will need the concept of harmonic extensions :
Deﬁnition K.2.
Denote by (cid:96) 2 p r G Latin Y t‹uq
the graph signal space C | r G Latin Yt‹u| equipped with the node weights t r µ g u g P r G Latin Yt‹u .
Given an arbitrary signal u P (cid:96) 2 p r G Latin Y t‹uq a harmonic extension of u to all of (cid:96) 2 p r G q is a signal u P (cid:96) 2 p r G q satisfying p ∆ r G u qp α q “ 0 @ α P r G Greek and u p h q “ u p h q @
h P r G Latin ď t‹u .
We ﬁrst note that the concept of harmonic extensions is both well-deﬁned an well-behaved:
Lemma K.3.
Fix u P (cid:96) 2 p r G Latin Y t‹uq .
There exists a unique harmonic extension u P (cid:96) 2 p r G q of u .
It is given as the solution to the convex optimization program min
E r G p
u
q subject to u p h q “ δ hg for all h P r G Latin ď t‹u .
Furthermore if u and v are the harmonic extensions of u and v , then p u ` v q is the (unique) harmonic extension of p u ` v q .
Proof.
We write a signal ψ P (cid:96) 2 p r G q as ψ “ p ψ, η q with ψ P (cid:96) 2 p r G Latin Y t‹uq and η P (cid:96) 2 p r G Greek q .
We then notice ψ “ argmin E r G p u q subject to ψ p h q “ ψ p h q for all h P r G Latin ď t‹u
[Equation]
P r G Greek and ψ p h q “ ψ p h q for all h P r G Latin
[Equation]
P
[Equation]
Greek and ψ p h q “ ψ p h q for all h P r G Latin ď t‹u ôp
[Equation]
Greek and ψ p h
[Equation]
“ ψ p
[Equation]
for all h P r G Latin ď t‹u .
Here, we treated η α and its complex conjugate as independent variables and used that E r G p¨q is a real-valued functional for the ﬁrst equivalence.
As harmonic extensions are thus equivalently characterised as the solutions of convex minimization programs, they are unique.
To prove the last statement, we note that by linearity of the graph Laplacian, p u ` v q certainly is a harmonic extension of p u ` v q .
Since harmonic extensions are unique, it is the only one.
After this preparatory effort, we are now ready to prove (15):
Lemma K.4.
For any δ ě 0 the signals t (cid:126)η δg u g P r G
[Equation]
Ť t‹u form a partition of unity of (cid:96) 2 p r G Greek q : ÿ
[Equation]
Equivalently we have
[Equation]
As an immediate Corollary we obtain
Corollary K.5.
For any δ ě 0 the signals t ψ
[Equation]
form a partition of unity of
[Equation]
p
[Equation]
q :
[Equation]
Proof.
Using the ’boundary conditions’ in (5), it is straightforward to verify that (16) is equivalent to (17).
From Lemma K.3 we now know that ψ δg , originally characterised as the solution of the problem
[Equation]
subject to u p
[Equation]
for all
[Equation]
is equivalently characterised as the harmonic extension of u p h q “ δ hg .
From the last statement of
Lemma K.3, we know that ř g P r G Latin Ť t‹u (cid:126)η δg is the unique harmonic extension of ÿ
[Equation]
But this – in turn – is the unique solution of the problem
[Equation]
Since we have
[Equation]
which is the lowest possible attainable value of E r G p¨q , and setting u “ 1 r G is compatible with the ’boundary condition’ u p h q “ 1 for all h P r G Latin Ť t‹u , we know that is the (unique) harmonic extension of 1 r G Latin Ť t‹u .
By the last statement of Lemma K.3 we thus have ÿ
[Equation]
Having established that we may write
[Equation]
together with the fact that every entry of each (cid:126)η δg is non-negative, we now know that
[Equation]
Furthermore – using our earlier estimate (13) – we now easily obtain
[Equation]
Hence – by positivity of the entries – we also have for each individual g P r G Latin that
[Equation]
For the weights t µ δg u g P G we then ﬁnd
[Equation]
if g ‰ ‹ .
We also write r µ p r G Greek q : “ ř α P r G Greek r µ α .
If g “ ‹ , we have
[Equation]
Having set the scene, we are now ready to prove Theorem 5.4.
Following Post & Simmer (2017), instead of checking the conditions of Deﬁnition 5.1 and Deﬁnition 5.2 it is instead sufﬁcient to check the following, with J r J as deﬁned in Section 5.2 to establish Theorem 5.6:
Lemma K.6.
In addition to identiﬁcation operators J, r J , assume that there exist additional operators
J 1 :
[Equation]
2 p G q Ñ (cid:96) 2 p r G q and r J 1 : (cid:96) 2 p r G q Ñ (cid:96) 2 p G q so that the following set of equations is satisﬁed with (cid:15)
[Equation]
Then the (normal) operators ∆ and r ∆ are (doubly) (-1)- ( (cid:15) “ 12 (cid:15) 1 ) -close with identiﬁcation-operator
J .
Here, we always have u P (cid:96) 2 p r G q and f P (cid:96) 2 p G q )
Proof.
This follows immediately after combining Proposition 4 .
12 with Theorem 4 .
15 of Post (2012).
We set J 1 f “ Jf and p r J 1 u qp x q “ u p x q and now determine the individual (cid:15) “ (cid:15) p δ q values for which these equations are satisﬁed:
Left
-hand-side of (18)
For the left hand side of (18) we note (using 2 ab ď a 2 ` b 2 and the fact that the ψ g form a partition of unity):
[Equation]
Here the second to last inequality follows from the deﬁnition of the weights µ δg .
Thus the left hand side of (18) holds with (cid:15) “ 0 .
Right
The right hand side of (18) holds trivially with (cid:15) “since we have chosen J ˚ “ r J .
Left-hand-side of (19):
Now let us check the l.h.s.
of (19).
We have:
[Equation]
Using the constant K deﬁned in (14) we have
[Equation]
if g ‰ ‹ .
We also write r µ p r G Greek q : “ ř
[Equation]
r µ α .
If G “ ‹ , we have
[Equation]
We next note
[Equation]
with Ă W x the vector with entries Ă W x p g q “ Ă W xg .
Thus for y ‰ ‹ we ﬁnd
[Equation]
We thus ﬁnd
[Equation]
ˇˇˇˇˇ
To bound the ﬁrst term of the estimate
[Equation]
we note (for y ‰ ‹ ) and δ small enough: ˆ
[Equation]
We also note (for y ‰ ‹ )
[Equation]
Thus we ﬁnd gfffe
[Equation]
[Table]
[Equation]
[Table]
[Equation]
To estimate the second term, we estimate
[Equation]
[Table]
[Equation]
to obtain ˇˇˇ ˇˇˇˇ ÿ g P
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
ˇˇˇ ˇˇˇˇ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
ˇˇˇˇˇˇˇ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
ˇˇˇˇˇˇˇ
Thus we ﬁnd (using that x (cid:126)η δg , (cid:126)η δy y
[Table]
2 p r G Greek q is a non-negative number and we have } ¨ } 2 ď } ¨ } 1 )
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Let us thus turn to the remaining term; corresponding to y “ ‹ : We have
[Equation]
We ﬁrst deal with the left summand.
We note
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
ˇˇˇˇˇˇ
[Equation]
[Table]
[Equation]
Thus
[Equation]
under the assumption δ ď 1 (implying δ 2 ď δ )
[Equation]
we have
[Equation]
This implies that we have
[Equation]
[Table]
Putting it all together
[Table]
we ﬁnd for δ ď 1 that
[Equation]
[Table]
with
[Equation]
Thus the left hand side of (19) holds with
[Equation]
Right-hand-side of (19):
Hence let us now check the right hand side of (19).
We note
[Equation]
Let us denote by M the matrix representation
[Equation]
We use the triangle inequality to arrive at
[Equation]
[Equation]
Using the fact that for g ‰ ‹ we have (cid:126)η δ g Ñ (cid:126) 0 an (cid:126)η 0 ‹ “ 1 r G Greek we ﬁnd in the ( δ Ñ 0 )-limit that
[Equation]
with
[Equation]
˚˝
[Equation] [Equation]
¨ ˚˝r
[Equation]
r
[Equation]
r
[Equation]
¨ ¨ ¨ r µ
[Equation]
r µ
[Equation]
r µ
[Equation]
¨ ¨ ¨ .
˛ ‹‚ acting on (cid:96) 2 p r G Greek Y t‹uq .
For any element v P (cid:96) 2 p r G q
[Equation]
let us denote its restriction to r G Greek Y t‹u by v P (cid:96) 2 p r G Greek Y t‹uq .
We thus ﬁnd
[Equation]
To proceed, we prove the following Lemma:
Lemma K.7.
Let i, j P r G Greek Y t‹u .
Denote by C r G Greek Yt‹u
p
i, j q the minimum number of edges for which ω ij ŋ 0 needed to connect i and j by a path.
Set
[Equation]
Furthermore set Ω : “
[Equation]
We have
[Equation]
We call C r G Greek Yt‹u the connectivity constant of the sub-graph r G Greek Y t‹u and note that it is well-deﬁned since we assume r G Greek Y t‹u to be connected.
Proof.
Fix i and j .
Let t i, g 1 , ..., g n , j u be the vertices traversed by a path of minimal length determining C r G
Greek Yt‹u p
i, j q .
We then have
[Equation]
With the help of this Lemma we then ﬁnd
[Equation]
To derive a bound for ›› M δ ´ M 0 ›› op in the second term of the estimate (23), we write
[Equation]
Here we denote by
[Equation]
the adjoint of the operator
[Equation]
Clearly } A } op “ } A : | op so that we have
[Equation]
To bound } B } op we note that B is diagonal and we have
[Equation]
so that }
[Equation]
To estimate } A } op we note
[Equation]
˛
[Equation]
We can consider the map
[Equation]
as a composition of maps
[Equation]
For the map Id :
[Equation]
Similarly we ﬁnd for the map Id : (cid:96) 2 p r G Latin q Ñ C | r G Latin | that } Id } op “ ˜ max g P r G Latin r µ g ¸ .
To bound the operator norm of the map A : C | r G Greek Yt‹u| Ñ C | r G Latin | , we use that the operator-norm is smaller than the maximal column-sum times b | r G Greek Y t‹u| .
Hence for A as a map from C | r G Greek Yt‹u| to C | r G Latin |
we ﬁnd
[Equation]
b
[Equation]
Greek
[Equation]
¨ ¨ ˚˝ min g P r G Latin µ δg ˛ ‹‚ ¨ max α P r G Greek » – ÿ a P r G Latin (cid:126)η δa p α q ﬁ ﬂ “ b | r G Greek Y t‹u| ¨ ¨ ˚˝ min g P r G Latin µ δg ˛ ‹‚ ¨ max α P r G Greek ” (cid:126)ζ δ ‹ p
α q ı “ δ ¨ K ¨ b | r G Greek Y t‹u| ¨ ¨ ˚˝ min g P r G Latin µ δg ¨ min α P r G Greek ar µ α ˛ ‹‚ ď δ ¨ K ¨ b | r G Greek Y t‹u|
¨ ¨ ˚˝ min g P r G Latin r µ g ¨ max α P r G Greek ar µ α ˛ ‹‚ .
Here we estimated max α P r G Greek ” (cid:126)ζ
[Equation]
‹ p
[Equation]
q ı ď min α P r G Greek ar µ α } (cid:126)ζ δ ‹ } (cid:96) 2 p r G Greek q .
In total, we ﬁnd for the operator-norm of
A :
[Equation]
p
[Equation]
G Greek Y
[Equation]
ÝÑ
[Equation]
r G Latin q .
that
[Equation]
˚˚˝ max g
[Equation]
µ g min g
[Equation]
G
[Equation]
max
[Equation] [Equation]
˛ ‹‹‚ .
Thus let us now investigate } D } op .
As before.
let us denote by u P (cid:96) 2 p r G Greek Y t‹uq
the restriction of an element u P (cid:96) 2 p r G to r G Greek Y t‹u .
We have } D } op “ ›››››
[Equation]
We note for the matrix representation of the ﬁrst term
[Equation]
that (with α, β P r G Greek Y t‹u )
we have
[Equation]
Using the ’maximal row sum trick’ complementary to the ’maximal column sum trick’ already used for A above and recalling the deﬁnition of the weights
[Equation]
we ﬁnd
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
It remains to bound the second term.
We
[Table]
ˇˇˇ
[Table] [Table]
ˇ ˇˇˇ ¨ ››
[Table]
[Equation]
´r
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Thus we ﬁnd
[Equation]
D
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In total, using (23) and (24)
[Equation]
we ﬁnd ›››
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
and may hence set
[Equation]
Left-hand-side of
The left hand side of (20) is true with (cid:15) “ 0 by deﬁnition.
Right-hand-side of
Let us thus check the right hand side of (20):
We have
[Equation]
We note
[Equation]
We ﬁrst deal with the left hand term of the estimate and note that for x “ ˚ we have
[Equation]
and in the limit δ Ñ
that
[Equation]
ˇ
[Equation] [Equation]
ˇˇ ˇˇˇ
[Equation]
ˇˇ ˇˇˇ
[Equation]
ˇˇˇˇˇˇ
[Equation]
Here we applied Lemma K.7.
Comparing the δ ą
and δ “ 0 terms, we ﬁnd
[Equation]
Thus we have ˇˇˇ
[Equation]
For the remaining term in (25) we note
[Equation]
[Table]
[Equation]
Equation (21):
It ﬁnally only remains to prove the energy differences of (21) and establish
[Equation]
We note that the (unique) operator associated to the energy E G via
[Equation]
is given by
[Equation]
Here the notation " y „ G x " signiﬁes that nodes x and y are connected within G through edges with positive edge-weights W xy ą 0 .
Similarly the operator associated to E r G via
[Equation]
is given by
[Equation]
with the equivalence relation „ r G precisely signifying that Ă W xy ą 0 .
As before.
let us denote by u P (cid:96) 2 p r G Greek Yt‹uq the restriction of an element u P (cid:96) 2 p r G to r G Greek Yt‹u .
We note
[Equation]
on the smaller graph G .
For the graph r G
we ﬁnd
[Equation]
Remembering that we have
[Equation]
and p
[Equation]
q
[Equation]
we note
[Equation]
Let us ﬁrst bound the terms corresponding to x ‰ ‹ : We have
[Equation]
as well as
[Equation]
Hence (for x ‰ ‹ )
[Equation]
For I x we ﬁnd – using Lemma K.7 – that
[Equation]
and hence ÿ
[Equation]
To bound | II x | we note
[Equation]
Thus we ﬁnd – using Cauchy-Schwarz – that ÿ
[Equation]
Here we denoted by r d α the degree of the node α .
We further note
[Equation]
Writing
[Equation]
for the sum of ’internal’ degrees of greek nodes within Greek Y t‹u at δ “ 1 and
[Equation]
external :
[Equation]
for the ’total connection strength’ between the Greek and Latin sector, we thus ﬁnd
[Equation]
It remains to bound the x “ ‹ term in (26).
To this end we note
[Equation]
and
[Equation]
For the difference of the energy forms we thus ﬁnd
[Equation]
We have ÿ
[Equation]
Greek
[Equation]
with the last term vanishing by symmetry.
This implies
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
Continuing, we ﬁnd
[Equation]
This – in turn – we can write as
E G p ψ ‹ , u q ´ E
[Equation]
G p ψ
[Equation]
, u q “ I ` II with
I : “ ÿ α P r G Greek (cid:126)ζ δ ‹ p
α q ÿ y „ Ă G α Ă W yα p u p α q ´ u p y qq , and
II : “ ÿ α P r G Greek ÿ y „ Ă G α y P r G Latin Ă W yα p u p‹q ´ u p α qq .
For the ﬁrst term, we ﬁnd
[Equation]
For the second term we note
[Equation]
L P ROOF OF T HEOREM 5.
We prove the following theorem:
Theorem L.1.
In the setting of Theorem 5.6 denote by T ( r T ) adjacency matrices or normalized graph Laplacians on (cid:96) 2 p G q ( (cid:96) 2 p G q ).
There are no functions η 1 , η 2 : r 0 , 1 s Ñ R ě 0 with η
i p
δ q Ñas δ Ñ
( i “ 1 , 2 ), families of identiﬁcation operators J δ , r J δ and ω P C so that J δ and r J δ are η 1 p δ q -quasi-unitary with respect to r T , T and ω while the operators r T and T remain ω - η 2 p
δ q close.
Proof.
We prove these two result through contradiction on a graph with two vertices and one edge with weight 1 { δ , which we collapse.
First ﬁx T ( r T ) to be the adjacency matrices
[Equation]
and
[Equation]
The eigenvectors and eigenvalues of Ă W are given by t´ 1 δ , 1 δ u and v ´ “ ˆ ´˙ and v ` “ ˆ ˙ .
Denote the orthogonal projections onto the corresponding eigenspaces by t P ´ , P ` u .
Take the function g to be deﬁned as
[Equation]
Then since g p 0 q “ 0 we have g p
[Equation]
Furthermore we have
[Equation]
We are interested in
[Equation]
Assuming
[Equation]
we also ﬁnd
[Equation]
Thus also
[Equation]
Taking the limit and using the condition }
J δ } op ď
[Equation]
we ﬁnd that › › J δ › › Ñ 0 as δ Ñ 0 .
Since we demand
[Equation]
q
[Equation]
with lim
[Equation]
we also ﬁnd } r J } op “ } r J ˚ } op Ñ 0 .
Next we note that we have
R
[Equation]
and demand
[Equation]
However
[Equation]
Thus we have our contradiction.
Hence let us now choose T ( r T ) as the normalized graph Laplacians associated to the adjacency matrices W ( Ă W ) from above.
We thus have
L
[Equation]
and
[Equation]
The eigenvectors and eigenvalues of Ă L are given by t 0 , 2 u and v 0 “ ˆ ˙ and v 2 “ ˆ ´˙ .
Denote the orthogonal projections onto the corresponding eigenspaces by t P 0 , P 2 u .
Then
[Equation]
Chose a function g such that g p 0 q “ 0 and without loss of generality assume g p 2 q “ 1 .
Then
[Equation]
Next we consider the demand
[Equation]
Since p Ă L ´ ωId q is bijective, (28) is implies
[Equation]
29) upon writing
[Equation]
We also write v “ ˆ
[Equation]
b ˙ .
We write r J
[Equation]
ˆ
[Equation]
b
[Equation]
˙ T and
J
[Equation]
˙ ` f p
[Equation]
q ¨ ˆ ˙ .
From (27), we know that lim
[Equation]
p δ
[Equation]
“ 0 , but we do not yet know the behaviour of f p¨q , a δ , b δ as δ Ñ 0 .
With the above notation, we ﬁnd from (29) that
[Equation]
Thus, combining this result with (29), we know that ››››ˆ
[Equation]
Thus, since both entries of the above vector need to tend to zero, we need both f p δ q ¨ a δ Ñ 1 and f p δ q ¨ b δ Ñas well as f p δ q ¨ a δ Ñ 0 and f p δ q ¨ b δ Ñwhich yields the desired contradiction.
M P ROOF OF T HEOREM 5.
We ﬁrst note how the graph Laplacian ∆ G N as we have deﬁned it, is consistent with the underlying positive (in the sense of non-negative eigenvalues)
Laplacian
[Equation]
on the unit circle S 1 .
To this end, ﬁx
ă h ăă 1 .
Fix a point x P S 1 .
For any suitable function f – by means of Taylor expansions – we may write
[Equation]
Adding these two terms, we ﬁnd
[Equation]
This motivates setting our edgeweights on G N to 1 { h 2 with h “ 2 π { N the distance between evenly spaced nodes on the unit-circle S 1 .
Remark M.1.
It should be noted that this consistency property – while given a heuristic to choose weights – does not (immediately) imply ’convergence’ of ∆ G N to ´ ∆ S 1 in the sense needed to e.g. apply Levie et al. (2019a).
As our proof of Theorem M proceeds completely without reference to the limit-circle, we do not proceed beyond the above heuristic in investigating in what (relevant) sense ∆ G N approximates ´ ∆ S 1 .
We thus now want to prove the following result:
Theorem M.2.
In the large graph setting of Section 5.2 choose all node-weights equal to one and
N to be odd for deﬁniteness.
There exists constants K 1 , K 2 “ O p 1 q so that for each N ě 1 , there exist identiﬁcation operators J, r J mapping between (cid:96) 2 p G N q and (cid:96) 2 p G N ` 1 q so that J and r J are p K 1 { N q -quasi-unitary with respect to ∆ G N , ∆ G N ` 1 and ω “ p´ 1 q .
Furthermore, the operators ∆ G N and ∆ G N ` 1 are p´ 1 q - p K 2 { N q close with identiﬁcation operator J .
Proof.
We ﬁrst note that the normalized eigenvectors of G N are given by
[Equation]
The corresponding eigenvalues are easily found to be
[Equation]
For deﬁniteness, we have assumed N to be odd, so that p N ` 1 q is even.
We deﬁne the identiﬁcation operator J : (cid:96) 2 p G N q Ñ (cid:96) 2 p G N ` 1 q via
J p φ Nk p x qq “ " φ N ` 1 k for K
ă Nφ N ` 1 k `for K
ă Non the orthonormal basis t φ Nk u 0 ď k ă N and extend it to all of (cid:96) 2 p G N q via normality.
This implies that precisely the eigenspace spanned by φ N ` 1 N ` 1(corresponding to the eigenvalue λ N ` 1 N ` 1“ p N ` 1 q 2 { π 2 ) does not lie in the image of J .
We set r J to be the adjoint J ˚ of J .
Choosing ω “ 1 , we shall now ﬁrst check the equations of Deﬁnition 5.1.
Since J is isometric, we have
[Equation]
as desired.
Since r J “ J ˚ , we have
[Equation]
Since r JJ “ Id (cid:96) 2 p G N q , what remains to be checked is the demand
[Equation]
We have
[Equation]
Thus let us now check that the conditions of Deﬁnition 5.2 are fulﬁlled.
We note that with our identiﬁcation operator and by symmetry ( λ Nk “ λ NN ´ k ), we have
[Equation]
π
[Equation]
sin
[Equation]
ˇˇˇˇˇ ˇ .
We now need to bound the right hand side uniformly in k as N Ñ 8 .
To this end we write a : “ 1 { N (which implies N ` 1 N “ 1 ` a ) and x “ kN (which for our allowed values of k implies 0 ď x ă 12 ).
With this we have
[Table]
[Equation]
Thus we are done if we can show that the function
[Equation]
ˇˇˇˇˇˇ is bounded on the rectangle r 0 , 1 s ˆ r 0 , 12 s .
We change variables y “ πx {p 1 ` a q and consider
[Equation]
ˇˇˇˇ on r 0 , 1 s ˆ r 0 , π 2 s instead.
Away from y “ 0 this is obvious.
Close to y “ 0 we might Taylor expand in numerators and denominators respectively and then (formally) divide them both respectively by y to see that the function F p a, y q is indeed regular at y “ 0 too and hence on the entire compact set r 0 , 1 s ˆ r 0 , π 2 s .
As a continuous function, F attains its supremum on this set.
Denote it by K .
Hence we now know
[Equation]
Thus we have established the desired O p 1 { N q -decay.
N P ROOF OF T HEOREM 6.
Theorem N.1.
For p ě 2 we have in the setting of Theorem 3.1 that } Ψ pN p f q ´ Ψ pN p h q} R K out ď ´ś
Nn
[Equation]
L n
[Equation]
n ¯ ¨ } f ´ h } L in .
In the setting of Theorem 4.3 or 5.4 and under the additional assumption that the ’ﬁnal’ identiﬁcation operator J N satisﬁes
[Equation] [Equation]
K
[Equation] [Equation]
p G N q for all f i P
[Equation]
p G N q , we have } Ψ
[Equation]
Proof.
To prove the ﬁrst claim, we note
[Equation]
where we used the reverse triangle inequality and the fact that } ¨ } (cid:96)
p
p r G out q ď } ¨ } (cid:96) 2 p r G out q for 2 ď p
[Equation]
To ﬁnish the proof we now only need to apply Theorem 3.1.
To prove the second claim we note
[Equation]
[Table]
and the claim follows as before.
The proof of the third claim proceed in complete analogy.
O A DDITIONAL DETAILS ON E XPERIMENTAL SETUP
Scaling Operators: The adjacency matrix fo the given graph is given by
A “ ¨ ˚˚˚˝   18          18     90 ˛ ‹‹‹‚ .
Collapsing Edges: We consider the setting introduced in Section 5.2 and consider a generic fully connected graph r G with | r G | “ 8 .
We consider a splitting into r G “ r G Latin Ť r G Greek Ť t‹u with | r G Latin | “ 3 and | r G Greek | “ 4 .
As described in Section 5.2, we assume Ą W ab , Ă W a ‹ “ O p 1 q , @ a, b P r
G Latin and Ă W αβ “ ω αβ δ and Ă W α ‹ “ ω α ‹ δ such that p ω αβ , ω
α ‹ “ O p 1 q for all α, β P r G Greek .
For completeness and reproducibility, the full adjacency matrix Ă W can be found in Appendix O. We set node weight on r G to one and – as discussed – construct a graph G with | G | “ 4 through ’collapsing strong edges’.
The adjacency matrix of the larger ’un-collapsed’ graph r G we consider in Section 7 is given as follows Ă W “ ¨ ˚˚˚˚˚˚˚˚˝                            16 { δ 7 { δ 18 { δ 19 { δ  12 16 { δ 6 { δ 22 { δ 3 { δ   7 { δ 6 { δ 1 { δ 90 { δ 10 14 18 { δ 22 { δ 1 { δ 23 { δ 11 15 19 { δ 3 { δ 90 { δ 23 { δ ˛ ‹‹‹‹‹‹‹‹‚ (31)
The exceptional vertex ‹ here carries index " 4 " (" ‹ “ 4 ").
Node weights are set to unity.
The Realm of Large Graphs: We also plot the difference in characteristic operators as opposed to their resolvents:
Their distances does not decay.
Experiments on Molecules: The dataset we consider is the QM 7 dataset, introduced in Blum &
Reymond (2009); Rupp et al.
This dataset contains descriptions of 7165 organic molecules, each with up to seven heavy atoms, with all non-hydrogen atoms being considered heavy.
A molecule is represented by its Coulomb matrix C Clmb , whose off-diagonal elements
[Equation]
|
[Equation]
| correspond to the Coulomb-repulsion between atoms i and j , while diagonal elements encode a polynomial ﬁt of atomic energies to nuclear charge Rupp et al.
[Equation]
For each atom in any given molecular graph, the individual Cartesian coordinates R i and the atomic charge Z i are also accessible individually.
To each molecule an atomization energy - calculated via density functional theory - is associated.
The objective is to predict this quantity, the performance metric is mean absolute error.
Numerically, atomization energies are negative numbers in the range ´ 600 to ´ 2200 .
The associated unit is r kcal/mol s .