Sign and Basis Invariant Networks for Spectral Graph Representation Learning
Abstract
We introduce SignNet and BasisNet—new neural architectures that are invariantto two key symmetries displayed by eigenvectors: (i) sign flips, since if v is aneigenvector then so is − v ; and (ii) more general basis symmetries, which occur inhigher dimensional eigenspaces with infinitely many choices of basis eigenvectors.
We prove that our networks are universal, i.e., they can approximate any continuous function of eigenvectors with the desired invariances.
Moreover, when usedwith Laplacian eigenvectors, our architectures are provably expressive for graphrepresentation learning: they can approximate any spectral graph convolution, cancompute spectral invariants that go beyond message passing neural networks, andcan provably simulate previously proposed graph positional encodings.
Experiments show the strength of our networks for molecular graph regression, learningexpressive graph representations, and learning neural fields on triangle meshes.
Introduction
Numerous machine learning models process eigenvectors, which arise in various scenarios includingprincipal component analysis, matrix factorizations, and operators associated to graphs or manifolds.
An important example is the use of Laplacian eigenvectors to encode information about the structureof a graph or manifold [Belkin and Niyogi, 2003, Von Luxburg, 2007, Lévy, 2006].
Positionalencodings that involve Laplacian eigenvectors have recently been used to generalize Transformersto graphs [Kreuzer et al., 2021, Dwivedi and Bresson, 2021] and to improve the expressive powerand empirical performance of graph neural networks (GNNs)
[Dwivedi et al., 2022].
Furthermore,these eigenvectors are crucial for defining spectral operations on graphs that are foundational to graphsignal processing and spectral GNNs [Ortega et al., 2018, Bruna et al., 2014].
However, there are nontrivial symmetries that should be accounted for when processing eigenvectors.
For instance, if v is an eigenvector, then so is − v , with the same eigenvalue.
More generally, if aneigenvalue has higher multiplicity, then there are infinitely many unit-norm eigenvectors that canbe chosen.
Indeed, a full set of orthonormal eigenvectors is only defined up to a change of basisin each eigenspace.
In the case of sign invariance, for any k eigenvectors there are 2 k possiblechoices of sign.
Accordingly, prior works randomly flip eigenvector signs during training in order toapproximately learn sign invariance [Kreuzer et al., 2021, Dwivedi et al., 2020].
However, learningall 2 k invariances is challenging and limits the effectiveness of Laplacian eigenvectors for encodingpositional information.
Sign invariance is a special case of basis invariance when all eigenvalues aredistinct, but general basis invariance is even more difficult to deal with.
In Appendix C.2, we showthat higher dimensional eigenspaces are abundant in real datasets; for instance, 64% of moleculegraphs in the ZINC dataset have a higher dimensional eigenspace.
In this work, we address the sign and basis ambiguity problems by developing new neural networks—
SignNet and BasisNet.
Our networks are universal and can approximate any continuous function of eigenvectors with the proper invariances.
Moreover, our networks are theoretically powerfulfor graph representation learning—they can approximate spectral graph convolutions and computepowerful spectral invariants, which allows our networks to express graph properties like subgraphcounts that message passing neural networks cannot.
Finally, Laplacian eigenvectors with SignNetand BasisNet can approximate many previously proposed graph positional encodings, including thosebased on random walks [Li et al., 2020, Dwivedi et al., 2022] and heat kernels [Mialon et al., 2021,
Feldman et al., 2022].
Experiments on molecular graph regression tasks, learning expressive graphrepresentations, and texture reconstruction on triangle meshes illustrate the empirical benefits of ourmodels’ approximation power and invariances.
Sign and Basis Invariant Networks
For an n × n symmetric matrix, let λ 1 ≤ . . .
≤λ n be the eigenvalues and v 1 , . .
. , v n the corresponding eigenvectors, which we may assumeto form an orthonormal basis.
For instance, wecould consider the normalized graph Laplacian
L = I − D − 1 / 2 AD − 1 / 2 , where A ∈ R n × nis the adjacency matrix and D is the diagonaldegree matrix of some underlying graph.
Forundirected graphs, L is symmetric.
Nonsymmetric matrices can be handled very similarly, as weshow in Appendix B.1.
Our goal is to parameterize a class of models f ( v 1 , . .
, v k ) taking keigenvectors as input in a manner that respectsthe eigenvector symmetries.
Sign invariance.
For any of the v i , the signflipped − v i is also an eigenvector, so a functionf : R n × k
→ R s (where s is an arbitrary outputdimension) should be sign invariant :
[Equation]
for all sign choices s i ∈ {− 1 , 1 } .
That is, wewant f to be invariant to the product group {− 1 , 1 } k .
This captures all eigenvector symmetries if theeigenvalues λ
i are distinct.
Basis invariance.
If the eigenvalues have higher multiplicity, then there are further symmetries.
Let V 1 , . .
, V l be bases of eigenspaces—i.e., V i = (cid:2) v i. . .
v i di (cid:3) ∈ R n × d
i has orthonormalcolumns and spans the eigenspace associated with the shared eigenvalue µ i
= λ
i 1 = . . .
= λ i di .
Any other orthonormal basis that spans the eigenspace is of the form V i Q for some orthogonal
Q ∈ O ( d i ) ⊆
R d i ×
d
i (see Appendix F.2).
Thus, a function f : R n × (cid:80) li = 1 d i → R s that is invariant tochanges of basis in each eigenspace satisfies
[Equation]
In other words, f is invariant to the product group
O ( d 1 ) × . . .
× O
( d l ) .
The number of eigenspacesl and the dimensions d i may vary between matrices; we account for this in Section 2.2.
As O (1) ={− 1 , 1 } , sign invariance is a special case of basis invariance when all eigenvalues are distinct.
Permutation equivariance.
For GNN models that output node features or node predictions, onetypically further desires f to be invariant or equivariant to permutations of nodes, i.e., along the entries(or rows) of each vector.
Thus, for f : R n × d → R n × d , we typically also require f ( PV 1 , . .
. , PV l ) =
Pf ( V 1 , . .
, V l ) for any permutation matrix P ∈ R n × n .
Figure 1 illustrates the full setup.
Graph Positional Encodings.
A major motivation for processing eigenvector input is for graphpositional encodings, which are additional features appended to each node in a graph that giveinformation about the position of that node in the graph.
These additional features are crucial forgeneralizing Transformers to graphs, and also have been found to improve performance of GNNs.
Figure 2 illustrates a standard pipeline and the use of our SignNet within it: the input adjacency, nodefeatures, and eigenvectors of a graph are used to compute a prediction about the graph. Laplacian eigenvectors are processed before being fed into this prediction model. Laplacian eigenvectorshave been widely used as positional encodings, and many works have noted that sign and/or basisinvariance must be dealt with in this case [Dwivedi and Bresson, 2021, Beaini et al., 2021, Dwivediet al., 2020, Kreuzer et al., 2021, Mialon et al., 2021, Dwivedi et al., 2022]. V
X
A
V—V 1 𝞺 𝜙 𝜙 , … ,
V k —V k 𝜙 𝜙
GraphAdjacency (n x n)
[Figure]
(n x d)
LaplacianEigvecs (n x
[Figure] [Figure] 2.1 Warmup: Neural Networks on One Eigenspace Before considering the general setting, we design neural networks that take a single eigenvector oreigenspace as input and are sign or basis invariant.
These single subspace architectures will becomebuilding blocks for the general architectures.
For one subspace, a sign invariant function is merely aneven function, and is easily parameterized.
Proposition 1.
A continuous function h : R n → R s is sign invariant if and only if
[Equation]
(3) for some continuous ϕ : R n → R s .
A continuous h : R n → R n is sign invariant and permutationequivariant if and only if (3) holds for a continuous permutation equivariant ϕ : R n → R n .
In practice, we parameterize ϕ by a neural network.
Any architecture choice will ensure signinvariance, while permutation equivariance can be achieved using elementwise MLPs (Multi-Layer Perceptrons), DeepSets [Zaheer et al., 2017], Transformers [Vaswani et al., 2017], or GNNs.
Next, we address basis invariance for a single d -dimensional subspace, i.e., we aim to parameterizemaps h : R n × d → R n that are (a) invariant to right multiplication by Q ∈ O
( d ) , and (b) equivariantto permutations along the row axis.
For (a), we use the mapping V
(cid:55)→ V V ⊤ from V to theorthogonal projector of its column space, which is O ( d ) invariant.
Mapping V
(cid:55)→ V V ⊤ does not loseinformation if we treat V as equivalent to V Q for any Q ∈ O ( d ) .
This is justified by the classicalfirst fundamental theorem of O ( d ) [Kraft and Procesi, 1996], which has recently been applied inmachine learning by Villar et al.
Regarding (b), permuting the rows of V permutes rows and columns of V V ⊤ ∈ R n × n .
Hence, wedesire the function ϕ : R n × n → R n on V V ⊤ to be equivariant to both row and column permutation:ϕ ( PV V ⊤ P ⊤ ) =
Pϕ ( V V ⊤ ) .
To parameterize such a mapping from matrices to vectors, we use aninvariant graph network ( IGN )
[Maron et al., 2018]—a neural network mapping to and from tensorsof arbitrary order R n d 1 → R n d 2 that has the desired permutation equivariance.
We thus parameterizea family with the requisite invariance and equivariance as follows:
[Equation]
Proposition 2 states that this architecture universally approximates O ( d ) invariant and permutationequivariant functions.
The full
Proposition 2.
Any continuous, O ( d ) invariant h : R n × d → R s is of the form h ( V ) =
ϕ ( V V ⊤ ) fora continuous ϕ .
For a compact domain Z ⊆ R n × d , maps of the form V (cid:55)→ IGN( V V ⊤ ) universallyapproximate continuous h : Z ⊆ R n × d → R n that are O ( d ) invariant and permutation equivariant.
2.2 Neural Networks on Multiple Eigenspaces
Next, we use the single-eigenspace models ϕ l ( V l ) as building blocks for a model on multipleeigenspaces.
To do so, we use functions of the form f ( V 1 , . .
, V l ) =
ρ ( ϕ 1 ( V 1 ) , . . .
, ϕ l ( V l )) , i.e., wefirst process each eigenspace individually with an invariant network, and then aggregate them via afunction ρ .
This approach is grounded in a general decomposition theorem for product spaces thatwe prove in Section A.
SignNet.
We parameterize our sign invariant network f : R n × k
→ R s on eigenvectors v 1 , . .
. , v k as
[Equation]
[Equation] where ϕ and ρ are unrestricted neural networks, and [ · ] i denotes concatenation of vectors.
The formϕ ( v i )+ ϕ ( − v i ) induces sign invariance for each eigenvector.
Since we do not yet impose permutationequivariance here, we term this model Unconstrained-SignNet .
To obtain a sign invariant and permutation equivariant f that outputs vectors in R n × s , we restrict ϕand ρ to be permutation equivariant networks from vectors to vectors, such as elementwise MLPs,
DeepSets [Zaheer et al., 2017], Transformers [Vaswani et al., 2017], or most standard GNNs.
Wename this permutation equivariant version SignNet .
If desired, we can additionally use eigenvalues λ iand node features X ∈ R n × q by adding them as arguments to ϕ :
[Equation]
BasisNet.
For basis invariance, let V i ∈ R n ×
d
i be an orthonormal basis of a d i dimensionaleigenspace.
Then we parameterize our Unconstrained-BasisNet f by
[Equation]
where each ϕ d i is shared amongst all subspaces of the same dimension d i , and l is the number ofeigenspaces (i.e., number of distinct eigenvalues, which can differ from the number of eigenvectorsk ).
As l differs between graphs, we may use zero-padding or a sequence model like a Transformer toparameterize ρ .
Again, ϕ d i and ρ are generally unrestricted neural networks.
To obtain permutationequivariance, we make ρ permutation equivariant and let ϕ d i =
IGN d
i : R n 2 → R n be IGNs frommatrices to vectors.
For efficiency, we will only use matrices and vectors in the IGNs (that is, notensors in R n p for p > 2 ), i.e., we use 2-IGN.
Our resulting BasisNet is
[Equation]
[Equation]
Expressive-BasisNet.
While we restrict SignNet to only use vectors and BasisNet to only use vectorsand matrices, higher order tensors are generally required for universally approximating permutationequivariant or invariant functions [Keriven and Peyré, 2019, Maron et al., 2019, Maehara and NT,2019].
Thus, we will consider a theoretically powerful but computationally impractical variant ofour model, in which we replace ρ and IGN d
i in BasisNet with IGNs of arbitrary tensor order.
Wecall this variant Expressive-BasisNet .
Universal approximation requires Ω( n n ) sized intermediatetensors [Ravanbakhsh, 2020].
We study Expressive-BasisNet due to its theoretical interest, and tojuxtapose with the computational efficiency and strong expressive power of SignNet and BasisNet.
For a summary of properties and more details about our models, see Appendix B.
In the multiple subspace case, we can prove universality of our models through a general decomposition theorem, which reduces the multiple subspace case to the single subspace case.
See Section Afor details; we have temporarily moved this Section in the revision due to space constraints, and wewill move this Section into the main paper in the camera-ready version.
Theoretical Power for Graph Representation Learning
Next, we establish that our SignNet and BasisNet can compute useful basis invariant and permutationequivariant functions on Laplacian eigenvectors for graph representation learning, including: spectral graph convolutions, spectral invariants, and existing graph positional encodings.
Expressive-BasisNetcan of course compute these functions, as it is universal, but this section shows that the practicalinvariant architectures SignNet and BasisNet can compute them as well.
3.1 SignNets and
BasisNets Generalize Spectral Graph Convolution
For node features X ∈ R n × q and an eigendecomposition V Λ V ⊤ , a spectral graph convolutiontakes the form f ( V, Λ , X )
= (cid:80) ni
=
θ i v i v ⊤ i X = V Diag( θ ) V ⊤ X , for some parameters
θ i , that mayoptionally be continuous functions
h ( λ i )
=
θ i of the eigenvalues [Bruna et al., 2014, Defferrard et al.
This family includes important functions like heat kernels and generalized PageRanks ongraphs [Li et al., 2019].
A spectral GNN is defined as multiple layers of spectral graph convolutionsand node-wise linear maps, e.g. V Diag( θ 2 ) V ⊤ σ (cid:0) V Diag( θ 1 ) V ⊤ XW 1 (cid:1)
W 2 is a two layer spectral
GNN. It can be seen (in Appendix H.1) that spectral graph convolutions are permutation equivariantand sign invariant, and if θ i
= h ( λ i ) (i.e. the spectral graph convolution is parametric) they areadditionally invariant to a change of bases in each eigenspace.
Our SignNet and BasisNet can be viewed as generalizations of spectral graph convolutions, as ournetworks can universally approximate all spectral graph convolutions of the above form.
For instance,
SignNet with ρ ( a 1 , . .
. , a k ) = (cid:80) k
i =1 a k and ϕ ( v i , λ i , X ) = {+1 
+} θ i v i v ⊤ i X directly yields the spectralgraph convolution.
This is captured in Theorem 1, which we prove in Appendix H.1.
In fact, we mayexpect SignNet to learn spectral graph convolutions well, according to the principle of algorithmicalignment [Xu et al., 2020]
(see Appendix H.1); this is supported by numerical experiments in
Appendix J.2, in which our networks outperform baselines in learning spectral graph convolutions.
Theorem 1.
SignNet universally approximates all spectral graph convolutions.
BasisNet universallyapproximates all parametric spectral graph convolutions.
In fact, SignNet and BasisNet are strictly stronger than spectral graph convolutions; there are functionscomputable by SignNet and BasisNet that cannot be approximated by spectral graph convolutionsor spectral GNNs.
One way to see this is through graph isomorphism power, as captured in thisnext result.
Proposition 3.
There exist infinitely many pairs of non-isomorphic graphs that SignNet and BasisNetcan distinguish, but spectral graph convolutions or spectral GNNs cannot distinguish.
3.2 BasisNets can Compute Spectral Invariants
Many works measure the expressive power of graph neural networks by comparing their power fortesting graph isomorphism
[Xu et al., 2019, Sato, 2020], or by comparing their ability to computecertain functions on graphs like subgraph counts [Chen et al., 2020, Tahmasebi et al., 2020].
Theseworks often compare GNNs to combinatorial invariants on graphs, especially the k -Weisfeiler-Lehman( k -WL) tests of graph isomorphism [Morris et al., 2021].
While we may also compare with these combinatorial invariants, as other GNN works that use spectralinformation have done [Beaini et al., 2021], we argue that it is more natural to analyze our networksin terms of spectral invariants , which are computed from the eigenvalues and eigenvectors of graphs.
There is a rich literature of spectral invariants from the fields of spectral graph theory and complexitytheory [Cvetkovi´c et al., 1997].
A spectral invariant must be invariant to permutations and changes ofbasis in each eigenspace, a characteristic shared by our networks.
The simplest spectral invariant is the multiset of eigenvalues, which we give as input to our networks.
Another widely studied, powerful spectral invariant is the collection of graph angles, which aredefined as the values α ij =
∥ V i V ⊤ i e j ∥ 2 , where V i ∈ R n × d
i is an orthonormal basis for the i thadjacency matrix eigenspace, and e j is the j th standard basis vector, which is zero besides a one inthe j th component.
These are easily computed by our networks (Appendix H.3), so our networksinherit the strength of these invariants.
We capture these results in the following theorem, which alsolists a few properties that graph angles determine [Cvetkovi´c, 1991].
Theorem 2.
BasisNet universally approximates the graph angles α ij .
The eigenvalues and graphangles (and thus BasisNet) can determine the number of length 3, 4, or 5 cycles, whether a graph isconnected, and the number of length k closed walks from any vertex to itself.
Relation to WL and message passing.
In contrast to this result, message passing GNNs are not ableto express any of these properties (see [Arvind et al., 2020, Garg et al., 2020] and Appendix H.3).
Although spectral invariants are strong, Fürer [2010] shows that the eigenvalues and graph angles—aswell as some strictly stronger spectral invariants—are not stronger than the 3-WL test (or, equivalently,the 2-Folklore-WL test).
Future work could study the combination of spectral invariants or spectralgraph positional encodings with combinatorial algorithms and graph neural networks.
3.3 SignNets and BasisNets Generalize Existing Graph Positional Encodings
Many graph positional encodings have been proposed, without any clear criteria on which to choosefor a particular task.
We prove (in Appendix H.2) that our efficient SignNet and BasisNet canuniversally approximate many previously used graph positional encodings, because we unify thesepositional encodings by expressing them as either a spectral graph convolution matrix or the diagonalof a spectral graph convolution matrix.
Proposition 4.
SignNet and BasisNet universally approximate node positional encodings based onheat kernels [Feldman et al., 2022] and random walks [Dwivedi et al., 2022].
BasisNet universallyapproximates diffusion and p -step random walk relative positional encodings [Mialon et al., 2021],and generalized PageRank and landing probability distance encodings [Li et al., 2020].
We note that diagonals of spectral convolutions are used as feature descriptors in the shape analysisliterature, such as the heat kernel signature [Sun et al., 2009] and wave kernel signature [Aubryet al., 2011].
In the language of recent works in graph machine learning, these are node positionalencodings computed from a discrete Laplacian of a triangle mesh.
This connection appears to beunnoticed in recent works on graph positional encodings.
Experiments
We demonstrate the strength of our networks in various experiments.
Appendix B shows simplepseudo-code and a diagram detailing the use of SignNet as a node positional encoding.
4.1 Graph Regression
[Table]
We study the effectiveness of SignNet for learning positional encodings (PEs) from the eigenvectorsof the graph Laplacian on the ZINC dataset of molecule graphs [Irwin et al., 2012] (using the [Table] subset of 12,000 graphs from Dwivedi et al.
We primarily consider three settings: 1) Nopositional encoding, 2) Laplacian PE (LapPE)—the k eigenvectors of the graph Laplacian withsmallest eigenvalues are concatenated with existing node features, 3) SignNet positional features—passing the eigenvectors through a SignNet and concatenating the output with node features.
Weparameterize SignNet by taking ϕ to be a GIN [Xu et al., 2019] and ρ to be an MLP.
We sum over ϕoutputs before the MLP when handling variable numbers of eigenvectors, so then the SignNet is ofthe form MLP (cid:16)(cid:80) li =1 ϕ ( v i ) + ϕ ( − v i ) (cid:17) (see Appendix K.2 for further details).
We consider fourdifferent base models that process the graph data and positional encodings: GatedGCN [Bresson and
Laurent, 2017], a Transformer with sparse attention only over neighbours [Kreuzer et al., 2021], PNA[Corso et al., 2020], and GIN [Xu et al., 2019] with edge features (i.e. GINE)
[Hu et al., 2020b].
Thetotal number of parameters of the SignNet and the base model is kept within a 500k budget.
Table 1 shows the results.
For all 4 base models, the PE learned with SignNet yields the best test MAE(mean absolute error) — lower MAE is better.
Notably, this includes the cases of PNA and GINE, forwhich Laplacian PE with simple random sign flipping was unable to improve performance over usingno PE at all.
Our best performing model is PNA base combined with SignNet, which achieves 0 .
084test MAE.
Besides SignNet, we consider two non-learned approaches to resolving eigenvector signambiguity—canonicalization and taking element-wise absolute values (see Appendix K.2 for details).
Results with GatedGCN show that these alternatives are not more effective than random sign flippingfor learning positional encodings.
We also consider an ablation of our SignNet architecture where weremove the sign invariance, using simply MLP([ ϕ ( v i )]
ki =1 ) .
Although the resulting architecture is nolonger sign invariant, ϕ still processes eigenvectors independently, meaning that only two invariances( ± 1 ) need be learned, significantly fewer than the 2 k total sign flip configurations.
Accordingly, thisnon-sign invariant learned positional encoding achieves a test MAE of 0 . 148 , improving over the
Laplacian PE ( 0 . 198 ) but falling short of the fully sign invariant SignNet ( 0 . 121 ).
In all cases, usingall available eigenvectors in SignNet significantly improves performance over using a fixed numberof eigenvectors.
In Appendix J.1, we also show that SignNet improves performance when no edgefeatures are included in the data.
These significant performance improvements from SignNet come with only a slightly higher computational cost.
For example, GatedGCN with no PE takes about 8.2 seconds per training iteration on
ZINC, while GatedGCN with 8 eigenvectors and SignNet takes about 10.6 seconds; this is only a29% increase in time, for a reduction of test MAE by over 50%.
Also, eigenvector computation timeis neglible, we need only precompute and save the eigenvectors once, and it only takes 15 seconds todo this for the 12,000 graphs of ZINC.
Comparison with SOTA.
In Table 2, we compare SignNet with state-of-the-art methods on graphlevel molecular regression tasks on ZINC (10,000 training graphs),
ZINC-full (about 250,000 graphs),and Alchemy
[Chen et al., 2019a] (10,000 training graphs).
We compare against both methods thatuse domain-specific knowledge about molecules, and domain-agnostic GNNs of various architectures.
We see that SignNet outperforms all domain-agnostic methods on ZINC and ZINC-full, and is withina standard deviation of the best domain-specific method.
Our mean score is the second best on
[Table]
Alchemy, and is within a standard deviation of the best.
We perform much better on ZINC (.084) thanother state-of-the-art positional encoding methods, like GNN-LSPE (.090)
[Dwivedi et al., 2022],
SAN (.139) [Kreuzer et al., 2021], and Graphormer (.122)
[Ying et al., 2021].
4.2 Counting Substructures and Regressing Graph Properties
[Figure]
Substructure counts (e.g. of cycles) and global graph properties (e.g. connectedness, diameter,radius) are important graph features that are known to be informative for problems in bio- andchemo-informatics
[Chen et al., 2020, Corso et al., 2020].
Following the setting of Zhao et al.
[2022],we show that SignNet with Laplacian positional encodings boosts the ability of simple GNNs tocount substructures and regress graph properties.
We take a 4-layer GIN as the base model for allsettings, and for SignNet we use GIN as ϕ and a Transformer as ρ to handle variable numbers ofeigenvectors (see Appendix K.4 for details). As shown in Figure 3, Laplacian PEs with sign-flip dataaugmentation improve performance for counting substructures but not for regressing graph properties,while Laplacian PEs processed by SignNet significantly boost performance on all tasks.
4.3 Neural Fields on Manifolds
Discrete approximations to the Laplace-Beltrami operator on manifolds have proven useful forprocessing data on surfaces, such as triangle meshes [Lévy, 2006].
Recently, Koestler et al.
[2022]propose intrinsic neural fields, which use eigenfunctions of the Laplace-Beltrami operator as positionalencodings for learning neural fields on manifolds.
For generalized eigenfunctions v 1 , . . .
, v k , ata point p on the surface, they parameterize functions f ( p ) =
MLP( v 1 ( p ) , . .
. , v k ( p )) .
As theseeigenfunctions have sign ambiguity, we use our SignNet to parameterize f ( p ) =
MLP( ρ ( [ ϕ ( v i ( p ))
+ϕ ( − v i ( p ))]
i =1 ,...,k ) ) , with ρ and ϕ being MLPs.
Table 3 shows our results for texture reconstruction experiments on all models from Koestler et al.[2022].
The total number of parameters in our SignNet-based model is kept below that of the originalmodel.
We see that the SignNet architecture improves over the original Intrinsic NF model and overother baselines — especially in the LPIPS (Learned Perceptual Image Patch Similarity) metric, whichhas been shown to be a typically better perceptual metric than PSNR or DSSIM [Zhang et al., 2018a].
While we have not yet tested this, we believe that SignNet would allow even better improvementswhen learning over eigenfunctions of different models, as it could improve transfer and generalization.
See Appendix D.1 for visualizations and Appendix K.5 for more details.
Eigvecϕ ( v 11 ) + ϕ ( − v 11 )
Eigvecϕ ( v 14 ) + ϕ ( − v 14 )
4.4 Visualization of Learned Positional Encodings
To better understand SignNet, we plot the first principal component of ϕ ( v ) + ϕ ( − v ) for twoeigenvectors on the cat model in Figure 4.
We see that SignNet encodes bilateral symmetry andstructural information on the cat model.
See Appendix D for plots of more eigenvectors and furtherdetails.
Related Work
In this section, we review selected related work.
A more thorough review is deferred to Appendix E.
Laplacian eigenvectors in GNNs.
Various recently proposed methods in graph deep learning havedirectly used Laplacian eigenvectors as node positional encodings that are input to a neural networkthat is, e.g., a message passing GNN [Dwivedi et al., 2020, 2022], or some variant of a Transformerthat is adapted to graphs [Dwivedi and Bresson, 2021, Kreuzer et al., 2021, Mialon et al., 2021,
Dwivedi et al., 2022].
None of these methods address basis invariance, and they only partially addresssign invariance for node positional encodings by randomly flipping eigenvector signs during training.
Graph positional encodings.
Other recent methods use positional encodings besides Laplacianeigenvectors.
These include positional encodings based on random walks [Dwivedi et al., 2022,
Mialon et al., 2021, Li et al., 2020], diffusion kernels on graphs [Mialon et al., 2021, Feldmanet al., 2022], shortest paths [Ying et al., 2021, Li et al., 2020], and unsupervised node embeddingmethods [Wang et al., 2022].
In particular, Wang et al.
[2022] use Laplacian eigenvectors for relativepositional encodings in an invariant way, but they focus on robustness, so they have stricter invariancesthat significantly reduce expressivity (see Appendix E.2 for more details).
These previously usedpositional encodings are mostly ad-hoc, less general since they can be provably expressed by SignNetand BasisNet (see Section 3.3), and/or are expensive to compute (e.g., all pairs shortest paths).
Conclusion and Discussion
SignNet and BasisNet are novel architectures for processing eigenvectors that are invariant to signflips and choices of eigenspace bases, respectively.
Both architectures are provably universal: theycan represent any continuous function with the corresponding invariances.
When used with Laplacianeigenvectors as inputs they can provably approximate spectral graph convolutions, spectral invariants,graph properties such as subgraph counts, and a number of other graph positional encodings.
Thesetheoretical results are supported by experiments showing that SignNet and BasisNet are highlyexpressive in practice, and learn effective graph positional encodings that improve the performanceof message passing graph neural networks.
Initial explorations show that SignNet and BasisNet canbe useful beyond graph representation learning, as eigenvectors are ubiquitous.
While we conduct experiments on graph machine learning tasks and a particular task on trianglemeshes, SignNet and BasisNet should also be applicable to processing eigenvectors in other settings,such as recommender systems and tasks in shape analysis.
We show significant empirical benefit inthe tasks that we consider, but we expect less benefit in cases where node features are sufficient to dowell on the task, or if the task does not require much sophisticated graph structure information tosolve.
Moreover, while we primarily consider eigenspaces in this work, sign invariance and basisinvariance applies to any model that processes subspaces of a vector space; future work may exploreour models on general subspaces.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We support the claims with theoretical and/or empirical
evidence.
(b) Did you describe the limitations of your work?
[Yes] We discuss some limitations in the conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix B.2.(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have read the guidelines; our paper conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We state all
assumptions either in the main text or in the appendix. (b) Did you include complete proofs of all theoretical results?
[Yes] We include all proofsin the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We includeexperimental code and instructions on the usage.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We try to give most experimental details in the main paper andappendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report standard deviations for multiple runs and/or
seeds for graph-level tasks, but not the texture reconstruction task.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix K.1.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the creators
of software libraries, data, and machine learning models.
(b) Did you mention the license of the assets?
[Yes] See Appendix K.1.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We provide our code.
We will also link to an open source version after anonymousreview.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Appendix K.1.(e)
Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] Yes, we discuss this in Appendix K.1; the data
most likely does not have personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A] No crowdsourcing or human subjects used.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
No crowdsourcing or human subjects
[Table]
used.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] No crowdsourcing or human subjects used.
[Table]
A Universality for Multiple Spaces
We have temporarily moved this section to the Appendix due to space constraints, we will move itback to the main paper in the camera-ready version. While the networks introduced in the Section 2.2 possess the desired invariances, it is not immediatelyobvious whether they are powerful enough to express all functions with these invariances. Theuniversality of our architectures follows as a corollary of the following general decomposition result,which may enable construction of universal architectures for other invariances as well.
Theorem 3 (Decomposition Theorem) . Let X 1 , . . . , X k be topological spaces, and let G i be agroup acting on X i for each i . We assume mild topological conditions on X i and G i hold.
For anycontinuous f : X = X 1 × . . .
× X k → R s that is invariant to the action of G =
G 1 × . . . × G k ,there exists continuous ϕ i and a continuous ρ : Z ⊆ R a → R s such that
[Equation] Furthermore: (1) each ϕ i can be taken to be invariant to G i , (2) the domain Z of ρ is compact if each X i is compact, (3) if X i = X j
and G i = G j , then ϕ i can be taken to be equal to ϕ j . This result says that when a product of groups G acts on a product of spaces X , for invariance to theproduct group G it suffices to individually process each smaller group G i on X i and then aggregatethe results. Along with the proof of Theorem 3, the mild topological assumptions are explained in
Appendix G.1.
The assumptions hold for sign invariance and basis invariance. By applying thistheorem, we can prove universality of our networks: Corollary 1.
Unconstrained-SignNet can represent any sign invariant function and UnconstrainedBasisNet can represent any basis invariant function. Expressive-BasisNet is a universal approximatorof functions that are both basis invariant and permutation equivariant. This result shows that Unconstrained-SignNet, Unconstrained-BasisNet, and Expressive-BasisNettake the correct functional form for their respective invariances (proofs in Appendix G.2). Notethat Expressive-BasisNet approximates all sign invariant functions as a special case, by treatingall inputs as one dimensional eigenspaces.
Accompanying the decomposition result, we show acorresponding universal approximation result (proof in Appendix G.3). Similarly to Theorem 3,the problem of approximating G = G 1 × . . . × G k invariant functions is reduced to approximatingseveral G i -invariant functions. B More Details on SignNet and BasisNet
[Table] [Table]
In Figure 2, we show a diagram that describes how SignNet is used as a node positional encodingfor a graph machine learning task.
In Table 4, we compare and contrast properties of the neuralarchitectures that we introduce.
In Figure 5, we give pseudo-code of SignNet for learning nodepositional encodings with a GNN prediction model.
B.1 Generalization Beyond Symmetric Matrices
In the main paper, we assume that the eigenspaces come from a symmetric matrix.
This holds for manycases of practical interest, as e.g. the Laplacian matrix of an undirected graph is symmetric.
However,we may also want to process directed graphs, or other data that have associated nonsymmetric matrices.
Our generalize a straightforward way handle nonsymmetric [Figure] matrices, as we detail here.
Let A ∈ R n × n be a matrix with a diagonalization A = V Λ V − 1 , whereΛ =
Diag( λ 1 , . . . , λ n ) contains the eigenvalues
λ i , and the columns of V =
[ v. . .
v n ] areeigenvectors.
Suppose we want to learn a function on the eigenvectors v 1 , . .
. , v k .
Unlike in thesymmetric matrix case, the eigenvectors are not necessarily orthonormal, and both the eigenvaluesand eigenvectors can be complex.
Real eigenvectors.
First, we assume the eigenvectors v i are all real vectors in R n .
We can take theeigenvectors to be real if A is symmetric, or if A has real eigenvalues (see Horn and Johnson [2012]
Theorem 1.3.29).
Also, suppose that we choose the real numbers R as our base field for the vectorspace in which eigenvectors lie.
Note that for any scaling factor c ∈ R \ { 0 } and eigenvector v ,we have that cv is an eigenvector of the same eigenvalue.
If the eigenvalues are distinct, then theeigenvectors of the form cv are the only other eigenvectors in the same eigenspace as v .
Thus, wewant a function to be invariant to scalings:
[Equation]
This can be handled by SignNet, by giving unit normalized vector inputs:
[Equation]
[Equation]
Now, say have bases of eigenspaces V 1 , . .
, V l with dimensions d 1 , . . .
, d l .
For a basis V i , we havethat any other basis of the same space can be obtained as V i W for some W ∈ GL R ( d i ) , the set ofreal invertible matrices in R d i × d i .
Indeed, the orthonormal projector for the space spanned by thecolumns of V i is given by V i ( V ⊤ i V i ) − 1 V ⊤ i .
Thus, if Z ∈ R n × d i is another basis for the columnspace of V i , we have that
V i ( V ⊤ i V i ) − 1 V ⊤ i
= Z ( Z ⊤ Z ) − 1 Z ⊤ , so
[Equation]
so let W =
( V ⊤ i V i ) − 1 V ⊤ i
Z ∈ R d i × d i .
Note that W is invertible, because it has inverse( Z ⊤ Z ) − 1 Z ⊤ V
i , so indeed V i W
=
Z for W ∈ GL R ( d i ) .
Thus, basis invariance in this case isof the form
[Equation]
Note that the distinct eigenvalue invariance is a special case of this invariance, as G R (1) =
R \ { 0 } .
We can again achieve this basis invariance by using a BasisNet, where the inputs to the ϕ d i areorthogonal projectors of the corresponding eigenspace:
[Equation]
Recall that if V i is an orthonormal basis, then the orthogonal projector is just V
i V ⊤ i , so this is a directgeneralization of BasisNet in the symmetric case.
Complex eigenvectors.
More generally, suppose V ∈ C n × n are complex eigenvectors, and we takethe base field of the vector space to be C .
The above arguments generalize to the complex case; inthe case of distinct eigenvalues, we want
[Equation]
However, this symmetry can not be as easily reduced to a unit normalization and a discrete signinvariance, as it can be in the real case.
Nonetheless, the basis invariant architecture directlygeneralizes, so we can handle the case of distinct eigenvalues by a more general basis invariantarchitecture as well.
The basis invariance is
[Equation]
The orthogonal projector of the image of V i is V
i
( V ∗ i V i ) − 1 V ∗ i , where there are now conjugatetransposes replacing the transposes.
Thus, BasisNet takes the form:
[Equation]
B.2 Broader Impacts
We believe that our models and future sign invariant or basis invariant networks could be useful in awide variety of applications.
As eigenvectors arise in many domains, it is difficult to predict the usesof these models.
We test on several molecular property prediction tasks, which have the potentialfor much positive impact, such as in drug discovery [Stokes et al., 2020].
However, recent workhas found that the same models that we use for finding beneficial drugs can also be used to designbiochemical weapons [Urbina et al., 2022].
Another major application of graph machine learningis in social network analysis, where positive (e.g. malicious node detection [Pandit et al., 2007])and negative (e.g. deanonymization [Narayanan and Shmatikov, 2009]) uses of machine learningare possible.
Even if there is no negative intent, bias in learned models can differentially impactparticular subgroups of people.
Thus, academia, industry, and policy makers must be aware of suchpotential negative uses, and work towards reducing the likelihood of them.
C More on Eigenvalue Multiplicities
In this section, we study the properties of eigenvalues and eigenvectors computed by numericalalgorithms on real-world data.
C.1 Sign and Basis Ambiguities in Numerical Eigensolvers
When processing real-world data, we use eigenvectors that are computed by numerical algorithms.
These algorithms return specific eigenvectors for each eigenspace, so there is some choice of signor basis of each eigenspace.
The general symmetric matrix eigensolvers numpy.linalg.eighand scipy.linalg.eigh both call LAPACK routines.
They both proceed as follows: for asymmetric matrix A , they first decompose it as A =
QTQ ⊤ for orthogonal Q and tridiagonal T , then they compute the eigendecomposition of T = W Λ W ⊤ , so the eigendecompositionof A is A = ( QW )Λ( W ⊤ Q ⊤ ) .
There are multiple ambiguities here: for diagonal sign matrices S
=
Diag( s 1 , . . .
, s n ) and S ′ =
Diag( s ′ 1 , . . .
, s ′ n ) , where s i , s ′ i ∈ {− 1 , 1 } , we have that
A = QS ( STS ) SQ ⊤ is also a valid tridiagonalization, as QS is still orthogonal,
SS = I , and STSis still tridiagonal.
Also, T = ( WS ′ )
Λ( S ′ W ⊤ ) is a valid eigendecomposition of T , as WS ′ is stillorthogonal.
In practice, we find that the general symmetric matrix eigensolvers numpy.linalg.eigh andscipy.linalg.eigh differ between frameworks but are consistent with the same framework.
Morespecifically, for a symmetric matrix A , we find that the eigenvectors computed with the defaultsettings in numpy tend to differ by a choice of sign or basis from those that are computed with thedefault settings in scipy.
On the other hand, the called LAPACK routines are deterministic, so theeigenvectors returned by numpy are the same in each call, and the eigenvectors returned by scipy arelikewise the same in each call. [Table]
Eigensolvers for sparse symmetric matrices like scipy.linalg.eigsh are required for large scaleproblems.
This function calls ARPACK, which uses an iterative method that starts with a randomlysampled initial vector.
Due to this stochasticity, the sign and basis of eigenvectors returned differsbetween each call.
Bro et al.
[2008] develops a data-dependent method to choose signs for each singular vector of asingular value decomposition.
Still, in the worst case the signs chosen will be arbitrary, and they donot handle basis ambiguities in higher dimensional eigenspaces.
Other works have made choicesof sign, such as by picking the sign so that the eigenvector’s entries are in the largest lexicographicorder [Tam and Dunson, 2022].
This choice of sign may work poorly for learning on graphs, as it issensitive to permutations on nodes.
For some graph regression experiments in Section 4.1, we try achoice of sign that is permutation invariant, but we find it to work poorly.
C.2 Higher Dimensional Eigenspaces in Real Graphs
Here, we investigate the normalized Laplacian eigenspace statistics of real-world graph data.
Forany graph that has distinct Laplacian eigenvalues, only sign invariance is required in processingeigenvectors.
However, we find that graph data tends to have higher multiplicity eigenvalues, so basisinvariance would be required for learning symmetry-respecting functions on eigenvectors.
Indeed, we show statistics for multi-graph datasets in Table 5 and for single-graph datasets with morenodes per graph in Table 6.
For multi-graph datasets, we consider :
• Molecule graphs: ZINC [Irwin et al., 2012, Dwivedi et al., 2020], ogbg-molhiv [Wu et al.
,2018, Hu et al.
, 2020a]• Social networks: IMDB-M, COLLAB [Yanardag and Vishwanathan, 2015, Morris et al.,2020a],•
Bioinformatics graphs: PROTEINS [Morris et al., 2020a]• Computer vision graphs: COIL-DEL
[Riesen and Bunke, 2008, Morris et al., 2020a].
For single-graph datasets, we consider:
• The 32 × 32 image grid as in Section J.2• Citation networks: Cora, Citeseer [Sen et al., 2008]• Co-purchasing graphs with Amazon Photo
[McAuley et al., 2015, Shchur et al., 2018].
We see that these datasets all contain higher multiplicity eigenspaces, so sign invariance is insufficientfor fully respecting symmetries.
The majority of graphs in each multi-graph dataset besides COILDEL contain higher multiplicity eigenspaces.
Also, the dimension of these eigenspaces can bequite large compared to the size of the graphs in the dataset.
The single-graph datasets have a largeproportion of their eigenvectors belonging to higher dimensional eigenspaces.
Thus, basis invariancemay play a large role in processing spectral information from these graph datasets. [Table]
C.3 Relationship to Graph Automorphisms
Higher multiplicity eigenspaces are related to automorphism symmetries in graphs.
For an adjacencymatrix A , the permutation matrix P is an automorphism of the graph associated to A if PAP ⊤ =
A .
If P is an automorphism, then for any eigenvector v of A with eigenvalue λ , we have
[Equation]
(18) so Pv is an eigenvector of A with the same eigenvalue λ .
If Pv and v are linearly independent, thenλ has a higher dimensional eigenspace.
Thus, under certain additional conditions, automorphismsymmetries of graphs lead to repeated eigenvalues [Sachs and Stiebitz, 1983, Teranishi, 2009].
C.4 Multiplicities in Random Graphs
It is known that almost all random graphs under the Erd ˝ os-Renyi model have no repeated eigenvaluesin the infinite number of nodes limit [Tao and Vu, 2017].
Likewise, almost all random graphsunder the Erd˝os-Renyi model are asymmetric in the sense of having no nontrivial automorphismsymmetries [Erdos and Rényi, 1963].
These results contrast sharply with the high eigenvaluemultiplicities that we see in real-world data in Section C.2.
Likewise, many types of real-world graphdata have been found to possess nontrivial automorphism symmetries [Ball and Geyer-Schulz, 2018].
This demonstrates a potential downside of using random graph models to study real-world data: theeigenspace dimensions and automorphism symmetries of random graphs may not agree with those ofreal-world data.
D Visualization of SignNet output
D.1 Cat Model Visualization
[Figure]
In Figure 6, we plot the eigenvectors of the cotangent Laplacian on a cat model, as well as the firstprincipal component of the corresponding learned ϕ ( v ) + ϕ ( − v ) from our SignNet model that wastrained on the texture reconstruction task.
Interestingly, this portion of our SignNet encodes bilateralsymmetry; for instance, while some eigenvectors differ between left feet and right feet, this portion ofour SignNet gives similar values for the left and right feet.
This is useful for the texture reconstructiontask, as the texture regression target has bilateral symmetry.
We also show principal components of outputs for the full SignNet model in Figure 7.
This is notas interpretable, as the outputs are high frequency and appear to be close to the texture that is theregression target.
If instead we trained the network on a task involving eigenvectors of multiplemodels, then we may expect the SignNet to learn more structurally interpretable mappings (as in thecase of the molecule tasks).
D.2 Molecule visualization
To better understand SignNet, in Figure 9 we visualize the learned positional encodings of a SignNetwith ϕ =
GIN, ρ = MLP (with a summation to handle variable eigenvector numbers) trained on
ZINC as in Section 4.1.
SignNet learns interesting structural information such as min-cuts (PC 3)and appendage atoms (PC 2) that qualitatively differ from any single eigenvector of the graph.
For this visualization we use a SignNet trained with a GatedGCN base model on ZINC, as in Section 4.1.
This SignNet uses GIN as ϕ and ρ as an MLP (with a sum before it to handle variablenumbers of eigenvectors), and takes in all eigenvectors of each graph.
See Figure 8 for all of theeigenvectors of fluorescein.
E More Related WorkE.1 Graph Positional Encodings
Various graph positional encodings have been proposed, which have been motivated for increasing expressive power or practical performance of graph neural networks, and for generalizing Transformersto graphs.
Positional encodings are related to so-called position-aware network embeddings [Chamiet al., 2020], which capture distances between nodes in graphs.
These include network embeddingmethods like Deepwalk [Perozzi et al., 2014] and node2vec [Grover and Leskovec, 2016], which havebeen recently integrated into GNNs that respect their invariances by Wang et al.
Further, Liet al.
[2020] studies the theoretical and practical benefits of incorporating distance features into graphneural networks.
Dwivedi et al.
[2022] proposes a method to inject learnable positional encodings intoeach layer of a graph neural network, and uses a simple random walk based node positional encoding.
You et al.
[2021] proposes a node positional encoding diag( A k ) , which captures the number ofclosed walks from a node to itself.
Dwivedi et al.
[2020] propose to use Laplacian eigenvectors aspositional encodings in graph neural networks, with sign ambiguities alleviated by sign flipping dataaugmentation.
Srinivasan and Ribeiro [2019] theoretically analyze node positional embeddings andstructural representations in graphs, and show that most-expressive structural representations containthe information of any node positional embedding.
While positional encodings in sequences as used for Transformers [Vaswani et al., 2017] are able toleverage the canonical order in sequences, there is no such useful canonical order for nodes in a graph,due in part to permutation symmetries.
Thus, different permutation equivariant positional encodingshave been proposed to help generalize Transformers to graphs.
Dwivedi and Bresson [2021] directlyadd in linearly projected Laplacian eigenvectors to node features before processing these featureswith a graph Transformer.
Kreuzer et al.
[2021] propose an architecture that uses attention over
Laplacian eigenvectors and eigenvalues to learn node or edge positional encodings.
Mialon et al.[2021] uses spectral kernels such as the diffusion kernel to define relative positional encodings thatmodulate the attention matrix.
Ying et al.
[2021] achieve state-of-the-art empirical performancewith simple Transformers that incorporate shortest-path based relative positional encodings.
Zhang
C
C C C C C
C O
O C C C C C C C O C C CC C C O O
Eigenvector−0.
−0. −0. C C C C
C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0.
−0.1 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0.
−0.1 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0.
−0.1 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector0.4 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector0.4 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0.
−0.1 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0.
−0.1 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0. −0.1 0. C
C C C C C C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0.
−0.1 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.3 −0.
−0.1 0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.
−0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C
C C C C C
C O
O C C C C C
C C O C C CC C C
O
O
Eigenvector−0.
−0.
et al. [2020] also utilize shortest-path distances for positional encodings Kim et al. [2021] develop higher-order transformers (that generalize invariant graph networks),which interestingly perform well on graph regression using sparse higher-order transformers withoutpositional encodings.
E.2 Eigenvector Symmetries in Graph Representation Learning
Many works that attempt to respect the invariances of eigenvectors solely focus on sign invariance(by using data augmentation)
[Dwivedi et al., 2020, Dwivedi and Bresson, 2021, Dwivedi et al., 2022,
Kreuzer et al., 2021].
This may be reasonable for continuous data, where eigenvalues of associatedmatrices may be usually distinct and separated (e.g. Puny et al.
[2022] finds that this empiricallyholds for covariance matrices of n -body problems).
However, discrete graph Laplacians are knownto have higher multiplicity eigenvalues in many cases, and in Appendix C.2 we find this to be true invarious types of real-world graph data.
Graphs without higher multiplicity eigenspaces are easierto deal with; in fact, graph isomorphism can be tested in polynomial time on graphs of bounded C C C C C C
C
O
O C C C C
C C C O C C C C C C
O
O
Eigenvector−0.
−0.
−0.
C C C C C C C
O
O C C C C
C C C O C C C C C C
O
O
Eigenvector−0.4 −0.
−0.2 −0.
C C C C C C C
O
O C C C C C C C O C C C C C C O O
Eigenvector−0.4 −0.
−0.2 −0.
C C C C C C C
O
O C C C C
C C C O C C C C C C
O
O
Eigenvector−0.
−0.
C C C C C C C
O
O C C C C
C C C
O C
C C C
C C
O
O
SignNet PC− −[-
-]
C
C C C C C
C O
O C C C C
C C C O C C C C C C
O
O
SignNet PC− −[-
-]
C
C C C C C
C O
O C C C C
C C C O C C C C C C
O
O
SignNet PC−
C
C C C C C
C O
O C C C C
C C C O C C C C C C
O
O
SignNet PC−1. −0.
multiplicity for adjacency matrix eigenvalues [Babai et al., 1982], with a time complexity that islower for graphs with lower maximum multiplicities.
A recent work of Wang et al.
[2022] proposes full orthogonal group invariance for functions thatprocess positional encodings.
In particular, for positional encodings Z ∈ R n × k , they parameterizefunctions f ( Z ) such that f ( Z ) =
f ( ZQ ) for all Q ∈ O ( k ) .
This indeed makes sense for networkembeddings like node2vec [Grover and Leskovec, 2016], as their objective functions are basedon inner products and are thus orthogonally invariant.
While they prove stability results whenenforcing full orthogonal invariance for eigenvectors, this is a very strict constraint compared toour basis invariance.
For instance, when k = n and all eigenvectors are used in V , the conditionf ( V ) =
f ( V Q ) implies that f is a constant function on orthogonal matrices, since any orthogonalmatrix W can be obtained as W = V Q for Q =
V ⊤ W ∈ O ( n ) .
In other words, for bases ofeigenspaces V 1 , . . .
, V l and V =
[ V. . .
V l ] , Wang et al.
enforces V Q ∼ =
V , while weenforce V Diag( Q 1 , . .
. , Q l ) ∼ = V .
While the columns of V Diag( Q 1 , . . . , Q l ) are still eigenvectors,the columns of V Q generally are not.
E.3 Graph Spectra and Learning on Graphs
More generally, graph spectra are widely used in analyzing graphs, and spectral graph theory [Chung,1997] studies the connection between graph properties and graph spectra.
Different graph kernelshave been defined based on graph spectra, which use robust and discriminative notions of generalizedspectral distance [Verma and Zhang, 2017], the spectral density of states [Huang et al., 2021], randomwalk return probabilities [Zhang et al., 2018b], or the trace of the heat kernel [Tsitsulin et al., 2018].
Graph signal processing relies on spectral operations to define Fourier transforms, frequencies,convolutions, and other useful concepts for processing data on graphs [Ortega et al., 2018].
Theclosely related spectral graph neural networks [Wu et al., 2020, Balcilar et al., 2020] parameterizeneural architectures that are based on similar spectral operations.
F Definitions, Notation, and BackgroundF.1 Basic Topology and Algebra Definitions
We will use some basic topology and algebra for our theoretical results.
A topological space ( X , τ )is a set X along with a family of subsets τ ⊆ 2 X satisfying certain properties, which gives usefulnotions like continuity and compactness.
From now on, we will omit mention of τ , and refer to a topological space as the set X itself.
For topological spaces X and Y , we write X ∼ =
Y and say that
X is homeomorphic to Y if there exists a continuous bijection with continuous inverse from X to
Y .
We will say X =
Y if the underlying sets and topologies are equal as sets (we will often use thisnotion of equality for simplicity, even though it can generally be substituted with homeomorphism).
For a function f : X → Y between topological spaces X and Y , the image im f is the set of valuesthat f takes, im f
= { f ( x ) : x ∈ X} .
This is also denoted f ( X ) .
A function f : X → Y is called atopological embedding if it is a homeomorphism from X to its image.
A group G is a set along with a multiplication operation G × G → G , such that multiplication isassociative, there is a multiplicative identity e ∈ G , and each g ∈ G has a multiplicative inverse g − 1 .
A topological group is a group that is also a topological space such that the multiplication and inverseoperations are continuous.
A group G may act on a set X by a function · : G × X → X .
We usually denote g · x as gx .
Atopological group is said to act continuously on a topological space X if · is continuous.
For anygroup G and topological space X , we define the coset
Gx = { gx : g ∈ G } , which can be viewed asan equivalance class of elements that can be transformed from one to another by a group element.
The quotient space X /G
=
{ Gx : x ∈ X} is the set of all such equivalence classes, with a topologyinduced by that of X .
The quotient map π : X → X /G is a surjective continuous map that sends xto its coset, π ( x )
=
Gx .
For x ∈ R s , ∥ x ∥ 2 denotes the standard Euclidean norm.
By the ∞ norm of functions f : Z → R sfrom a compact Z to a Euclidean space R s , we mean
[Equation]
F.2 Background on Eigenspace Invariances
Let V =
[ v. . .
v d ] and W = [ w. . .
w d ] ∈ R n × d be two orthonormal bases for the samed dimensional subspace of R n .
Since V and W span the same space, their orthogonal projectorsare the same, so V V ⊤ =
WW ⊤ .
Also, since V and W have orthonormal columns, we have
V ⊤ V = W ⊤ W = I ∈
R d × d .
Define Q = V ⊤ W .
Then Q is orthogonal because
[Equation]
Moreover, we have that
[Equation]
Thus, for any orthonormal bases V and W of the same subspace, there exists an orthogonal Q ∈ O ( d )such that V Q = W .
For another perspective on this, define the Grassmannian Gr( d, n ) as the smooth manifold consistingof all d dimensional subspaces of R n .
Further define the Stiefel manifold St( d, n ) as the setof all orthonormal tuples [ v. . .
v d ] ∈ R n × d of d vectors in R n .
Letting O ( d ) act by rightmultiplication, it holds that St( d, n ) /O
( d ) ∼ =
Gr( d, n ) .
This implies that any O ( d ) invariant functionon St( d, n ) can be viewed as a function on subspaces.
See e.g. Gallier and Quaintance [2020]
Chapter5 for more information on this.
We will use this relationship in our proofs of universal representation.
When we consider permutation invariance or equivariance, the permutation acts on dimensions of sizen .
Then a tensor X ∈ R n k × d is called an order k tensor with respect to this permutation symmetry,where order 0 are called scalars, order 1 tensors are called vectors, and order 2 tensors are calledmatrices.
Note that this does not depend on d ; in this work, we only ever consider vectors and scalarswith respect to the O ( d ) action.
G Proofs of Universality
We begin by proving the two propositions for the single subspace case from Section 2.1.
Proposition 1.
A continuous function h : R n → R s is sign invariant if and only if
[Equation]
(3) for some continuous ϕ : R n → R s .
A continuous h : R n → R n is sign invariant and permutationequivariant if and only if (3) holds for a continuous permutation equivariant ϕ : R n → R n .
Proof.
If h ( v ) =
ϕ ( v ) + ϕ ( − v ) , then h is obviously sign invariant.
On the other hand, if h is signinvariant, then letting ϕ ( v ) =
h ( v ) / 2 gives that h ( v ) =
ϕ ( v ) + ϕ ( − v ) , and ϕ is of course continuous.
If h ( v ) =
ϕ ( v ) + ϕ ( − v ) for a permutation equivariant ϕ , then h ( − Pv ) =
ϕ ( − Pv ) + ϕ ( Pv ) =
Pϕ ( − v ) + Pϕ ( v ) =
P ( ϕ ( v ) + ϕ ( − v ))
=
Ph ( v ) , so h is permutation equivariant and sign invariant.
If h is permutation equivariant and sign invariant, then define ϕ ( v )
=
h ( v ) / 2 again; it is clear that ϕis continuous and permutation equivariant.
Proposition 2.
Any continuous, O ( d ) invariant h : R n × d → R s is of the form h ( V ) =
ϕ ( V V ⊤ ) fora continuous ϕ .
For a compact domain Z ⊆ R n × d , maps of the form V (cid:55)→ IGN( V V ⊤ ) universallyapproximate continuous functions
h : Z ⊆ R n × d → R n that are O ( d ) invariant and permutationequivariant.
Proof.
The case without permutation equivariance holds by the First Fundamental Theorem of O ( d )(Lemma 2).
For the permutation equivariant case, let Z ′
=
{ V V ⊤ : V ∈ Z} and let ϵ > 0 .
Note that Z ′is compact, as it is the continuous image of a compact set.
Since h is O ( d ) invariant, the firstfundamental theorem of O ( d ) shows that there exists a continuous function ϕ :
Z ′ ⊆
R n × n → R nsuch that h ( V ) =
ϕ ( V V ⊤ ) .
Since h is permutation equivariant, for any permutation matrix P wehave that
[Equation]
[Equation]
(22) so ϕ is a continuous permutation equivariant function from matrices to vectors.
Then note that Kerivenand Peyré [2019] show that invariant graph networks (of generally high tensor order in hidden layers)universally approximate continuous permutation equivariant functions from matrices to vectors oncompact sets of matrices.
Thus, an IGN can ϵ -approximate ϕ , and hence V (cid:55)→ IGN( V V ⊤ )
canϵ -approximate h .
G.1 Proof of Decomposition Theorem
[Equation]
Here, we give the formal statement of Theorem 3, which provides the necessary topological assumptions for the theorem to hold.
In particular, we only require the G i be a topological group that actscontinuously on X i for each i , and that there exists a topological embedding of each quotient spaceinto some Euclidean space.
That the group action is continuous is a very mild assumption, and itholds for any finite or compact matrix group, which all of the invariances we consider in this papercan be represented as.
A topological embedding of the quotient space into a Euclidean space is desired, as we know how toparameterize neural networks with Euclidean outputs and inputs, whereas dealing with a quotient space is generally difficult.
Many different conditions can guarantee existence of such an embedding.
For instance, if the quotient space is a smooth manifold, then the Whitney Embedding Theorem(Lemma 5) guarantees such an embedding.
Also, if the base space X i is a Euclidean space and G i isa finite or compact matrix Lie group, then a map built from G -invariant polynomials gives such anembedding (González and de Salas [2003] Lemma 11.13).
Figure 10 provides a commutative diagram representing the constructions in our proof.
Theorem 3 (Decomposition Theorem) .
Let X 1 , . . .
, X k be topological spaces, and let G i be atopological group acting continuously on X i for each i .
Assume that there is a topological embeddingψ i : X i /G i
→ R a i of each quotient space into a Euclidean space R a i for some dimension a i .
Then, for any continuous function f : X = X 1 × . . .
×
X k
→ R s that is invariant to the action of
G = G 1 × . . .
× G k , there exists continuous functions ϕ
i : X i → R a i and a continuous functionρ : Z ⊆ R a → R s , where a = (cid:80) i a i such that
[Equation]
Furthermore: (1) each ϕ i can be taken to be invariant to G i , (2) the domain Z is compact if each X iis compact, (3) if X i =
X j
and G i =
G j , then ϕ i can be taken to be equal to ϕ j .
Proof.
Let π i :
X i → X i /G
i denote the quotient map for X i /G i .
Since each G i acts continuously,
Lemma 3 gives that the quotient of the product space is the product of the quotient spaces, i.e. that( X 1 × . . . × X k ) / ( G 1 × . . .
G
[Equation] [Equation]
and the corresponding quotient map π : X /G is given by
[Equation]
By passing to the quotient (Lemma 1), there exists a continuous ˜ f : X /G
→ R s on the quotientspace such that f = ˜ f ◦ π .
By Lemma 4, each X i /G
i is compact if X i is compact.
Defining theimage
Z i =
ψ
i ( X i /G i ) ⊆
R a i , we thus know that Z i is compact if X i is compact.
Moreover, as ψ i is a topological embedding, it has a continuous inverse ψ − 1 i on its image Z i .
Further,we have a topological embedding ψ : X /G
→ Z = Z 1 × . . .
×
Z k given by ψ = ψ 1 × . . .
× ψ k ,with continuous inverse ψ − 1 = ψ − 1 1 × . . .
× ψ − 1 k .
Note that
[Equation]
So we define
[Equation]
[Equation]
[Equation]
[Equation]
Thus, f
= ρ ◦ ϕ
=
ρ ◦ ( ϕ 1 × . . .
× ϕ k ) , so equation (9) holds.
Moreover, the ρ and ϕ i are continuous,as they are compositions of continuous functions.
Furthermore, (1) holds as each ϕ
i is invariantto G i because each π i is invariant to G i .
Since each Z i is compact if X i is compact, the product
Z = Z 1 × . . .
× Z k is compact if each X i is compact, thus proving (2).
To show the last statement (3), note simply that if X i =
X j
and G i =
G j , then the quotient maps areequal, i.e. π i =
π j .
Moreover, we can choose the embeddings to be equal, so say ψ i =
ψ j .
Then,ϕ i =
ψ i ◦ π i =
ψ j ◦ π j = ϕ j , so we are done.
G.2 Universality of SignNet and BasisNet
Here, we prove Corollary 1 on the universal representation and approximation capabilities of our
Unconstrained-SignNets, Unconstrained-BasisNets, and Expressive-BasisNets.
We proceed in several steps, first proving universal representation of continuous functions when we do not requirepermutation equivariance, then proving universal approximation when we do require permutationequivariance.
G.2.1 Sign Invariant Universal Representation
Recall that S n − 1 denotes the unit sphere in R n .
As we normalize eigenvectors to unit norm, thedomain of our functions on k eigenvectors are on the compact space ( S n − 1 ) k .
Corollary 2 (Universal Representation for SignNet) .
A continuous function f : ( S n − 1 ) k
→ R s issign invariant, i.e. f ( s 1 v 1 , . . .
, s k v k ) =
f ( v 1 , . .
, v k ) for any s i ∈ {− 1 , 1 } , if and only if thereexists a continuous ϕ : R n → R 2 n − 2 and a continuous ρ : R (2 n − 2) k
→ R s such that
[Equation]
[Equation]
Proof.
It can be directly seen that any f of the above form is sign invariant.
Thus, we show that any sign invariant f can be expressed in the above form.
First, we show thatwe can apply the general Theorem 3.
The group G i = { 1 , − 1 } acts continuously and satisfies that 1052 S n − 1 / { 1 , − 1 } = RP n − 1 , where RP n − 1 is the real projective space of dimension n − 1 .
Since
RP n − 1 is a smooth manifold of dimension n − 1 , Whitney’s embedding theorem states that thereexists a (smooth) topological embedding ψ i : RP n − 1 → R 2 n − 2 (Lemma 5).
Thus, we can apply the general theorem to see that f = ρ ◦ ˜ ϕ
k for some continuous ρ and ˜ ϕ k .
Notethat each ˜ ϕ
i =
˜ ϕ is the same, as each X i =
S n − 1 and G i = { 1 , − 1 } is the same.
Also, Theorem 3says that we may assume that ˜ ϕ is sign invariant, so ˜ ϕ
( x ) =
˜ ϕ
( − x ) .
Letting ϕ
( x ) =
˜ ϕ
( x ) / 2 , we aredone with the proof.
G.2.2 Sign Invariant Universal Representation with Extra Features
Recall that we may want our sign invariant functions to process other data besides eigenvectors, suchas eigenvalues or node features associated to a graph.
Here, we show universal representation forwhen we have this other data that does not possess sign symmetry.
The proof is a simple extension of
Corollary 2, but we provide the technical details for completeness.
Corollary 3 (Universal Representation for SignNet with features) .
For a compact space of featuresΩ ⊆
R d , let f ( v 1 , . .
. , v k , x 1 , . . .
, x k ) be a continuous function f : ( S n − 1 × Ω) k
→ R s .
Then f is sign invariant for the inputs on the sphere, i.e.
[Equation]
if and only if there exists a continuous ψ : R n + d → R 2 n − 2+ d and a continuous ρ : R (2 n − 2+ d ) k
→ R ssuch that
[Equation]
Proof.
Once again, the sign invariance of any f in the above form is clear.
We follow very similar steps to the proof of Corollary 2 to show that we may apply Theorem 3.
Wecan view Ω as a quotient space, after quotienting by the trivial group that does nothing, Ω ∼ = Ω / { 1 } .
The corresponding quotient map is id Ω , the identity map.
Also, Ω trivially topologically embeds in 1073 R d by the inclusion map.
As G i = {− 1 , 1 } × { 1 } acts continuously, by Lemma 3 we have that [Equation]
[Equation]
with corresponding quotient map π × id Ω , where π is the quotient map to RP n − 1 .
Letting ˜ ψ be the embedding of RP n − 1 → R 2 n − 2 guaranteed by Whitney’s embedding theorem(Lemma 5), we have that ψ = ˜
ψ
× id Ω is an embedding of RP n − 1 × Ω
→ R 2 n − 2+ d .
Thus, we canapply Theorem 3 to write f = ρ ◦ ˜ ϕ
k for ˜ ϕ
= ( ˜ ψ × id Ω ) ◦ ( π × id Ω ) , so
[Equation]
34) where ˜ ϕ ( v i , x i ) =
˜ ϕ
( − v i , x i ) .
Letting ϕ
( v i , x i ) =
˜ ϕ
( v i , x i ) / 2 , we are done.
G.2.3 Basis Invariant Universal Representation
Recall that St( d, n ) is the Stiefel manifold of d -tuples of vectors ( v 1 , . .
. , v d ) where v i ∈
R nand v 1 , . .
. , v d are orthonormal.
This is where our inputs lie, as our eigenvectors are unit norm andorthogonal.
We will also make use of the Grassmannian Gr( d, n ) , which consists of all d -dimensionalsubspaces in R n .
This is because the Grassmannian is the quotient space for the group action wewant, Gr( d, n ) ∼ =
St( d, n ) /O
( d ) , where Q ∈ O ( d ) acts on V ∈ St( d, n ) ⊆
R n × d by mapping V to
V Q [Gallier and Quaintance, 2020].
Corollary 4 (Universal Representation for BasisNet) .
For dimensions d 1 , . . .
, d l ≤ n let f be acontinuous function on St( d 1 , n ) × . . .
×
St( d l , n ) .
Further assume that f is invariant to O ( d 1 ) ×. . .
× O
( d l ) , where O ( d i ) acts on St( d i , n ) by multiplication on the right.
Then there exist continuous ρ : R (cid:80) li =1 2 d i ( n − d i )
→ R s and continuous ϕ
i : St( d i , n ) → R 2 d i ( n − d i )such that
[Equation]
35) where the ϕ i are O
( d i ) invariant functions, and we can take ϕ i = ϕ
j if d i =
d j .
Proof.
Letting X i
=
St( d i , n ) and G i
= O ( d i ) , it can be seen that G i acts continuously on X i .
Also,we have that the quotient space St( d i , n ) /O
( d i ) =
Gr( d i , n ) is the Grassmannian of d i dimensionalsubspaces in R n , which is a smooth manifold of dimension
d i ( n − d i ) .
Thus, the Whitney embeddingtheorem (Lemma 5) gives a topological embedding ψ
i : Gr( d i , n ) → R 2 d i ( n − d i ) .
Hence, we may apply Theorem 3 to obtain continuous O
( d i )
invariant ϕ
i : St( d i , n ) → R 2 d i ( n − d i )and continuous ρ : R (cid:80) li =
1 2 d i ( n − d i )
→ R s , such that f = ρ ◦ ( ϕ 1 × . . .
× ϕ l ) .
Also, if d i =
d j , then
X i =
X j
and G i =
G j , so we can take ϕ i = ϕ j .
G.2.4 Basis Invariant and Permutation Equivariant Universal Approximation
With the restriction that f ( V 1 , . . .
, V l ) : R n × (cid:80) i d i → R n be permutation equivariant and basis invariant, we need to use the impractically expensive Expressive-BasisNet to approximate f .
Universalityof permutation invariant or equivariant functions from matrices to scalars or matrices to vectors isdifficult to achieve in a computationally tractable manner [Maron et al., 2019, Keriven and Peyré,2019, Maehara and NT, 2019].
One intuitive reason to expect this is that universally approximatingsuch functions allows solution of the graph isomorphism problem [Chen et al., 2019b], which is acomputationally difficult problem.
While we have exact representation of basis invariant functionsby continuous ρ and ϕ i when there is no permutation equivariance constraint, we can only achieveapproximation up to an arbitrary ϵ > 0 when we require permutation equivariance.
Corollary 5 (Universal Approximation for Expressive-BasisNets) .
Let f ( V 1 , . .
, V l ) :
St( d 1 , n ) ×. . .
×
St( d l , n ) → R n be continuous, O ( d 1 ) × . . .
× O ( d l ) invariant, and permutation equivariant.
Then f can be ϵ -approximated by an Expressive-BasisNet.
Proof.
By invariance, Corollary 4 of the decomposition theorem shows that f can be written as
[Equation]
36) for some continuous O ( d i ) invariant φ d i and continuous ρ .
By the first fundamental theorem of O ( d )(Lemma 2), each φ d i can be written as φ d
i
( V i ) =
ϕ
d i ( V i V ⊤ i ) for some continuous ϕ d i .
Let
[Equation]
37) which is compact as it is the image of the compact space
St( d 1 , n ) × . . .
×
St( d l , n ) under a continuousfunction.
Define h : Z ⊆ R n 2 × l → R n by
[Equation]
Then note that h is continuous and permutation equivariant from matrices to vectors, so it can beϵ -approximated by an invariant graph network [Keriven and Peyré, 2019], call it (cid:103) IGN .
If we define˜ ρ = (cid:103) IGN and IGN d i ( V i V ⊤ i ) =
V i
V ⊤ i (this identity operation is linear and permutation equivariant,so it can be exactly expressed by an IGN), then we have ϵ -approximation of f by(cid:103) IGN [Equation] 1 V ⊤ 1 , . . . , V l V ⊤ l ) =
˜ ρ
[Equation]
IGN d 1 ( V 1 V ⊤ 1 ) , . . .
, IGN d l
[Equation]
l V ⊤ l [Equation] [Equation] G.3 Proof of Universal Approximation for General Decompositions
Theorem 4.
Consider the same setup as Theorem 3, where X i are also compact.
Let Φ i be afamily of G i -invariant functions that universally approximate G i -invariant continuous functions
X i
→ R a i , and let R be a set of continuous function that universally approximate continuousfunctions Z ⊆ R a → R s for every compact Z , where a = (cid:80) i a i .
Then for any ε > 0 and any
G -invariant continuous function f : X 1 × . . .
×
X k
→ R s there exists ϕ ∈ Φ and ρ ∈ R such that∥ f − ρ ( ϕ 1 , . .
. , ϕ k )
∥ ∞ < ε .
Proof.
Consider a particular G -invariant continuous function f : X 1 × . . .
×
X k
→ R s .
By
Theorem 3 there exists G i -invariant continuous functions ϕ
i : X i → R a i and a continuous functionρ ′ :
Z ⊆ R a → R s (where a = (cid:80) i a i ) such that
[Equation]
Now fix an ε > 0 .
For any ρ ∈ R and any ϕ i ∈
Φ i ( i = 1 , . . .
k ) we may bound the difference fromf as follows (suppressing the v
i ’s for brevity),
[Equation]
. , ϕ ′ k )
− ρ ( ϕ 1 , . .
. , ϕ k ) ∥ ∞ =
I + II
Now let K ′
= (cid:81) ki
=
im ϕ ′
i .
Since each ϕ
i is continuous and defined on a compact set
X i we knowthat
im ϕ
i is compact, and so the product K is also compact.
Since K ′ is compact, it is contained in aclosed ball B ( r ) of radius r > 0 centered at the origin.
Let K be the closed ball B ( r + 1) of radiusr + 1 centered at the origin, so K contains K ′ and a ball of radius 1 around each point of K ′ .
Wemay extend ρ ′ continuously to K as needed, so assume ρ ′ :
K → R s .
By universality of R we maypick a particular ρ : K → R s , ρ ∈ R such that
[Equation]
Keeping this choice of ρ , it remains only to bound II.
As ρ is continuous on a compact domain, itis in fact uniformly continuous.
Thus, we can choose a δ ′ > 0 such that if ∥ y − z ∥ 2 ≤ δ ′ , then∥ ρ ( y ) − ρ ( z ) ∥ ∞ < ϵ/ 2 , and then we define δ = min( δ ′ , 1) .
Since Φ i universally approximates ϕ ′
i we may pick ϕ i ∈
Φ i such that ∥ ϕ
i − ϕ ′ i
∥ ∞ < δ/ √ k , andthus ∥ ( ϕ 1 , . .
. , ϕ k ) − ( ϕ ′ 1 , . . .
ϕ ′ k ) ∥ ∞ ≤ δ .
With this choice of ϕ i , we know that (cid:81) ki
=
im ϕ
i ⊆
K(because each ϕ
i ( x i ) is within distance 1 of ϕ
i ( x i ) ).
Thus, ρ ( ϕ 1 ( x 1 ) , . . . , ϕ k ( x k )) is well-defined,and we have
[Equation]
< ε/due to our choice of δ , which completes the proof.
H Basis Invariance for Graph Representation Learning
H.1 Spectral Graph Convolution
In this section, we consider spectral graph convolutions, which for node features X ∈ R n × q take theform f
( V, Λ , X ) =
(cid:80) ni
=
θ i v i v ⊤
i X for some parameters θ i .
We can optionally take θ i
= h ( λ i ) forsome continuous function h :
R → R of the eigenvalues.
This form captures most popular spectralgraph convolutions in the literature [Bruna et al., 2014, Hamilton, 2020, Bronstein et al., 2017]; often,such convolutions are parameterized by taking h to be some analytic function such as a simple affinefunction
[Kipf and Welling, 2017], a linear combination in a polynomial basis [Defferrard et al.,
2016, Chien et al., 2021], or a parameterization of rational functions [Levie et al., 2018, Bianchi et al.
First, it is well known and easy to see that spectral graph convolutions are permutation equivariant, asfor a permutation matrix P we have
[Equation]
Also, it is easy to see that they are sign invariant, as ( − v i )( − v i )
⊤
= v i v ⊤ i .
However, if the θ i do notdepend on the eigenvalues, then the spectral graph convolution is not necessarily basis invariant.
Forinstance, if v 1 and v 2 are in the same eigenspace, and we change basis by permuting v ′
1 = v 2 andv ′
2 = v 1 , then if θ 1 ̸
= θ 2 the spectral graph convolution will generally change as well.
On the other hand, if θ i = h ( λ i ) for some function h : R → R , then the spectral graph convolutionis basis invariant.
This is because if v i and v j belong to the same eigenspace, then λ i = λ
j soh ( λ i )
=
h ( λ j ) .
Thus, if v i 1 , . .
. , v i d are eigenvectors of the same eigenspace with eigenvalue λ ,we have that (cid:80) dl
=1 h ( λ i l ) v i l v ⊤ i l = h ( λ ) (cid:80) dl
=1 v i l v ⊤ i l .
Now, note that (cid:80) dl
=1 v i l v ⊤ i l is the orthogonalprojector onto the eigenspace [Trefethen and Bau III, 1997].
A change of basis does not change thisorthogonal projector, so such spectral graph convolutions are basis invariant.
Another way to see this basis invariance is with a simple computation.
Let V 1 , . .
, V l be theeigenspaces of dimension d 1 , . . .
, d l , where V i ∈ R n × d i .
Let the corresponding eigenvalues beµ 1 , . . .
, µ l .
Then for any orthogonal matrices Q i ∈ O
( d i ) , we have
[Equation]
so the spectral graph convolution is invariant to substituting V j Q j for V j .
Now, we give the proof that shows SignNet and BasisNet can universally approximate spectral graphconvolutions.
Theorem 1 (Learning Spectral Graph Convolutions) .
Suppose the node features X ∈ R n × q takevalues in compact sets.
Then SignNet can universally approximate any spectral graph convolution,and both BasisNet and Expressive-BasisNet can universally approximate any parametric spectralgraph convolution.
Proof.
Note that eigenvectors and eigenvalues of normalized Laplacian matrices take values incompact sets, since the eigenvalues are in [0 , 2] and we take eigenvectors to have unit-norm.
Thus,the whole domain of the spectral graph convolution is compact.
Let ε > 0 .
First, consider a spectral graph convolution f ( V, Λ , X ) =
(cid:80) ni
=
θ i v i v ⊤
i X .
For SignNet,let ϕ ( v i , λ i , X ) approximate the function ˜ ϕ
( v i , λ i , X ) =
θ i v i v ⊤
i X to within ε/n error, which
DeepSets can do since this is a continuous permutation equivariant function from vectors to vectors[Segol and Lipman, 2019]
(note that we can pass λ
i as a vector in R n by instead passing λ i 1 , where1 is the all ones vector).
Then ρ = (cid:80) ni
=1 is a linear permutation equivariant operation that canbe exactly expressed by DeepSets, so the total error is within ε .
The same argument applies whenθ i
= h ( λ i ) for some continuous function h .
For the basis invariant case, consider a parametric spectral graph convolution f ( V, Λ , X ) =
1193 (cid:80) ni
=1 h ( λ i ) v i v ⊤ i X .
Note that if the eigenspace bases are V 1 , . .
, V l with eigenvalues µ 1 , . . . , µ l , wecan write the f ( V, Λ , X )
=
(cid:80) li =1 h ( µ j ) V j V ⊤ j X .
Again, we will let ρ = (cid:80) li
=1 be a sum function,which can be expressed exactly by DeepSets.
Thus, it suffices to show that h ( µ j ) V j V ⊤
j X can be ϵ/napproximated by a 2-IGN (i.e. an IGN that only uses vectors and matrices).
Note that since h is continuous, we can use an elementwise MLP (which IGNs can learn) toapproximate f 1 ( µ 11 ⊤ , V V ⊤ , X ) =
( h ( µ ) 11 ⊤ , V V ⊤ , X ) to arbitrary precision (note that we represent the eigenvalue µ as a constant matrix µ 11 ⊤ ).
Also, since a 2-IGN can learn matrix vectormultiplication (Cai and Wang [2022] Lemma 10), we can approximate f 2 ( h ( µ ) 11 ⊤ , V V ⊤ , X ) =( h ( µ ) 11 ⊤ , V V ⊤ X ) , as V i V ⊤ i ∈ R n 2 is a matrix and X ∈ R n × q is a vector with respect to permutation symmetries.
Finally, we use an elementwise MLP to approximate the scalar-vector multiplicationf 3 ( h ( µ ) 11 ⊤ , V V ⊤ , X )
=
h ( µ ) V V ⊤ X .
Since f 3 ◦ f 2 ◦ f 1 ( µ 11 ⊤ , V V ⊤ , X ) = h ( µ ) V V ⊤ X , andsince 2-IGNs universally approximate each f i , applying Lemma 6 shows that a 2-IGN can approximate h ( µ ) V V ⊤ X to ϵ/n accuracy, so we are done.
Since Expressive-BasisNet is stronger than
BasisNet, it can also universally approximate these functions.
From the proof, we can see that SignNet and BasisNet need only learn simple functions for the ρ andϕ when h is simple, or when the filter is non-parametric and we need only learn θ i .
Xu et al.
[2020]propose the principle of algorithmic alignment, and show that if separate modules of a neural networkeach need only learn simple functions (that is, functions that are well-approximated by low-orderpolynomials with small coefficients), then the network may be more sample efficient.
If we do notrequire permutation equivariance, and parameterize SignNet and BasisNet with simple MLPs, thenalgorithmic alignment may suggest that our models are sample efficient.
Indeed, ρ = (cid:80) is a simplelinear function with coefficients 1 , and ϕ ( V, λ, X ) =
h ( λ ) V V ⊤ X is quadratic in V and linear in X ,so it is simple if h is simple.
Proposition 3. There exist infinitely many pairs of non-isomorphic graphs that SignNet and BasisNetcan distinguish, but spectral graph convolutions or spectral GNNs cannot distinguish. Proof. The idea is as follows: we will take graphs G and give them the node feature matrix X G =
D 1 / 2 1 , i.e. each node has as feature the square root of its degree.
Then any spectral graph convolution(or, the first layer of any spectral GNN) will map V Diag( θ ) V ⊤ X to something that only depends onthe degree sequence and number of nodes.
Thus, any spectral graph convolution or spectral GNNwill have the same output (up to permutation) for any such graphs G with node features X G and thesame number of nodes and same degree sequence.
On the other hand, SignNet and BasisNet candistinguish between infinitely many pairs of graphs ( G (1) , G (2) ) with node features ( X G (1) , X G (2) )and the same number of nodes and degree sequence; this is because SignNet and BasisNet can tellwhen a graph is bipartite.
For each n ≥ 5 , we will define G (1) and G (2) as connected graphs with n nodes, with the samedegree sequence.
Also, we define G (1) to have node features X (1) i
= (cid:113) d (1) i , where d (1) i is the degreeof node i in G (1) , and similarly G (2)
has node features X (2) i = (cid:113) d (2) i .
Now, note that X (1) is aneigenvector of the normalized Laplacian of G (1) , and it has eigenvalue 0 .
As we take the eigenvectorsto be orthonormal (since the normalized Laplacian is symmetric), for any spectral graph convolutionwe have that
[Equation]
Where D 1 is the diagonal degree matrix of G (1) .
Likewise, any spectral graph convolution outputsθ 1 (cid:80) j
( d (2) j ) D 1 / 2 2 1 for G (2) .
Since D 1 and D 2 are the same up to a permutation, we have that anyspectral graph convolution has the same output for G (1) and G (2) , up to a permutation.
In fact, thisalso holds for spectral GNNs, as the first layer will always have the same output (up to a permutation)on G (1) and G (2) , so the latter layers will also have the same output up to a permutation.
Now, we concretely define G (1) and G (2) .
This is illustrated in Figure 11 and Figure 12.
For n = 5 ,let G (1) contain a triangle with nodes w 1 , w 2 , w 3 , and have a path of length 2 coming out of one ofthe nodes in the triangle, say w 1 connects to w 4 , and w 4 connects to w 5 .
This is not bipartite, as thereis a triangle.
Let G (2) be a bipartite graph that has 2 nodes on the left ( v 1 , v 2 ) and 3 nodes on theright ( v 3 , v 4 , v 5 ) .
Connect v 1 with all nodes on the right, and connect v 2 with v 3 and v 4 .
Note that both G (1) and G (2) have the same number of nodes and the same degree sequence{ 3 , 2 , 2 , 2 , 1 } .
Thus, spectral graph convolutions or spectral GNNs cannot distinguish them.
How- wwwww
G (1) vvvvv
G (2) wwwwww
G (1) vvvvv 5 v
G (2) ever, SignNet and BasisNet can distinguish them, as they can tell whether a graph is bipartite bychecking the highest eigenvalue of the normalized Laplacian.
This is because the multiplicity ofthe eigenvalue 2 is the number of bipartite components.
In particular, SignNet can approximatethe function ϕ
( v i , λ i , X ) =
λ
i and ρ ≈ max ni =1 .
Likewise, BasisNet can approximate the functionϕ
d i ( V i V ⊤ i , λ i ) =
λ
i and ρ ≈ max li =1 .
This in fact gives an infinite family of graphs that SignNet / BasisNet can distinguish, but spectralgraph convolutions or spectral graph GNNs cannot.
To see why, suppose we have G (1) and G (2) forsome n ≥ 5 .
Then we construct a pair of graphs on n + 1 nodes with the same degree sequence.
Todo this, we add another node to the path of G (1) , thus giving it degree sequence { 3 , 2 , . . .
For
G (2) , we add a node v n +1 to the side that v n is not contained on (e.g. for n = 5 , we add v 6 to the leftside, as v 5 was on the right), then connect v n to v n +1 to also give a degree sequence { 3 , 2 , . .
Note that the non-bipartiteness of G (1) and bipartiteness of G (2) are preserved.
H.2 Existing Positional Encodings
Here, we show that our SignNets and BasisNets universally approximate various types of existinggraph positional encodings.
The key is to show that these positional encodings are related to spectralgraph convolution matrices and the diagonals of these matrices, and to show that our networks canapproximate these matrices and diagonals.
Proposition 5.
If the eigenvalues take values in a compact set, SignNets and BasisNets universally approximate the diagonal of any spectral graph convolution matrix f
( V, Λ) =
diag (cid:0)(cid:80) ni
=1 h ( λ i ) v i v ⊤ i (cid:1) .
BasisNets can additionally universally approximate any spectral graph convolution matrix f
( V, Λ) = 1266 (cid:80) n i =1 h ( λ i ) v i v ⊤ i .
Proof.
Note that the v i come from a compact set as they are of unit norm.
The λ i are from a compactset by assumption; this assumption holds for the normalized Laplacian, as λ i ∈ [0 , 2] .
Also, as diagis linear, the spectral graph convolution diagonal can be written (cid:80) ni
=1 h ( λ i )diag( v i v ⊤ i ) .
Let ϵ > 0 .
For SignNet, let ρ = (cid:80) ni =1 , which can be exactly expressed as it is a permutationequivariant linear operation from vectors to vectors.
Then ϕ ( v i , λ i ) can approximate the functionλ i diag( v i v ⊤ i ) to arbitrary precision, as it is a permutation equivariant function from vectors tovectors [Segol and Lipman, 2019].
Thus, letting ϕ approximate the function to ϵ/n accuracy, SignNetcan approximate f to ϵ accuracy.
Let l be the number of eigenspaces V 1 , . .
, V l , so f ( V, Λ) =
(cid:80) li =1 h ( µ i ) V
i V ⊤ i .
For BasisNet, weneed only show that it can approximate the spectral graph convolution matrix to ϵ/l accuracy, as a2-IGN can exactly express the diag function in each ϕ d
i , since it is a linear permutation equivariantfunction from matrices to vectors.
A 2-IGN can universally approximate the function f 1 ( µ i , V
i V ⊤ i ) =( h ( µ i ) ,
V i V ⊤ i ) , as it can express any elementwise MLP.
Also, a 2-IGN can universally approximatethe scalar-matrix multiplication f 2 ( h ( µ i ) , V i V ⊤ i )
=
h ( µ i ) V
i V ⊤ i by another elementwise MLP.
Since h ( µ i ) V i V ⊤ i =
f 2 ◦ f 1 ( µ i , V i V ⊤ i )
, Lemma 6 shows that a single 2-IGN can approximate thiscomposition to ϵ/l accuracy, so we are done.
Proposition 4.
SignNet and BasisNet universally approximate node positional encodings based onheat kernels [Feldman et al., 2022] and random walks [Dwivedi et al., 2022].
BasisNet universallyapproximates diffusion and p -step random walk relative positional encodings [Mialon et al., 2021],and generalized PageRank and landing probability distance encodings [Li et al., 2020].
Proof.
We will show that we can apply the above Proposition 5, by showing that all of thesepositional encodings are spectral graph convolutions.
The heat kernel embeddings are of the formdiag (cid:0)(cid:80) ni =1 exp( − tλ i ) v i v ⊤ i (cid:1) for some choices of the parameter t , so they can be approximated by
SignNets or BasisNets.
Also, the diffusion kernel [Mialon et al., 2021] is just the matrix of thisheat kernel, and the p -step random walk kernel is (cid:80) ni =1 (1 − γλ i ) p v i v
⊤ i for some parameter γ , so
BasisNets can universally approximate both of these.
For the other positional encodings, we let v i be the eigenvectors of the random walk Laplacian
I − D − 1 A instead of the normalized Laplacian
I − D − 1 / 2 AD − 1 / 2 .
The eigenvalues of these two
Laplacians are the same, and if ˜ v i is an eigenvector of the normalized Laplacian
then D − 1 / 2 ˜ v
i is aneigenvector of the random walk Laplacian with the same eigenvalue [Von Luxburg, 2007].
Then with v i as the eigenvectors of the random walk Laplacian, the random walk positional encodings(RWPE) in Dwivedi et al.
[2022] take the form
[Equation]
for any choices of integer k .
The distance encodings proposed in Li et al.
[2020] take the form
[Equation]
46) for some function f 3 .
We restrict to continuous f 3 here; shortest path distances can be obtained by adiscontinuous f 3 that we discuss below.
Their generalized PageRank based distance encodings canbe obtained by
[Equation]
for some γ k ∈ R , so this is a spectral graph convolution.
They also define so-called landing probabilitybased positional encodings, which take the form
[Equation]
for some choices of integer k .
Thus, BasisNets can approximate these distance encoding matrices.
Another powerful class of positional encodings is based on shortest path distances between nodesin the graph [Ying et al., 2021, Li et al., 2020].
Shortest path distances can be expressed in a form similar to the spectral graph convolution, but require a highly discontinuous function.
If wedefine f 3 ( x 1 , . . . ,
x n ) =
min
i : x i ̸ =
i to be the lowest index such that x i is nonzero, then we canwrite the shortest path distance matrix as f 3 ( D − 1 A, ( D − 1 A ) 2 , . . .
, ( D − 1 A ) n ) , where f 3 is appliedelementwise to return an n × n matrix.
As ( D − 1 A ) k
= (cid:80) ni
=1 (1 − λ i )
k
v i v ⊤ i , BasisNets can learnthe inside arguments, but cannot learn the discontinuous function f 3 .
H.3 Spectral Invariants
Here, we consider the graph angles α ij =
∥ V i V ⊤ i e j ∥ 2 , for i = 1 , . . .
, l where l is the number ofeigenspaces, and j = 1 , . . . , n .
It is clear that graph angles are permutation equivariant and basisinvariant.
These graph angles have been extensively studied, so we cite a number of interestingproperties of them.
That graph angles determine the number of length 3, 4 and 5 cycles, theconnectivity of a graph, and the number of length k closed walks is all shown in Chapter 4 of Cvetkovi´cet al.
Other properties may be of use for graph representation learning as well.
For instance,the eigenvalues of node-deleted subgraphs of a graph G are determined by the eigenvalues and graphangles of G ; this may be useful in extending recent graph neural networks that are motivated by nodedeletion and the reconstruction conjecture [Cotta et al., 2021, Bevilacqua et al., 2022, Papp et al.
,2021, Tahmasebi et al., 2020].
Now, we prove that BasisNet can universally approximate the graph angles.
The graph properties weconsider in the theorem are all integer valued (e.g. the number of cycles of length 3 in a graph is aninteger).
Thus, any two graphs that differ in these properties will differ by at least 1, so as long aswe have approximation to ε < 1 / 2 , we can distinguish any two graphs that differ in these properties.
Recall the statement of Theorem 2.
Theorem 2.
BasisNet can universally approximate the graph angles α ij .
The eigenvalues and graphangles (and thus BasisNets) can determine the number of length 3, 4, and 5 cycles, whether a graphis connected, and the number of length k closed walks from any vertex to itself.
Proof.
Note that the graph angles satisfy
[Equation]
where V i is a basis for the i th adjacency matrix eigenspace, and
e ⊤ j V i V ⊤ i
e j is the ( j, j ) -entry of V
i V ⊤ i .
These graph angles are just the elementwise square roots of the diagonals of the matrices
V
i V ⊤ i .
As f 1 ( V i V ⊤ i ) =
diag( V i V ⊤ i ) is a permutation equivariant linear function from matrices to vectors,2-IGN on V
i V ⊤ i can exactly compute this with 0 error.
Then a 2-IGN can learn an elementwise
MLP to approximate the elementwise square root f 2 (diag( V i V ⊤ i ))
= (cid:112) diag( V i V ⊤ i ) to arbitraryprecision.
Finally, there may be remaining operations f 3 that are permutation invariant or permutationequivariant from vectors to vectors; for instance, the α ij are typically gathered into a matrix of sizel × n where the columns are lexicographically sorted ( l is the number of eigenspaces)
[Cvetkovi´cet al., 1997], or we may have a permutation invariant readout to compute a subgraph count.
A
DeepSets can approximate f 3 without any higher order tensors besides vectors [Zaheer et al., 2017,
Segol and Lipman, 2019].
As 2-IGNs can approximate each f i individually, a single 2-IGN can approximate f 3 ◦ f 2 ◦ f 1 by
Lemma 6.
Also, since the graph properties considered in the theorem are integer-valued, BasisNetcan distinguish any two graphs that differ in one of these properties.
To see that message passing graph neural networks (MPNNs) cannot determine these quantities, weuse the fact that MPNNs cannot distinguish between two graphs that have the same number of nodesand where each node (in both graphs) has the same degree.
For k ≥ 3 , let C k denote the cycle graphof size k , and C k + C k denote the graph that is the union of two disjoint cycle graphs of size k .
MPNNs cannot distinguish between C 2 k and C k + C k for k ≥ 3 , because they have the same numberof nodes, and each node has degree 2.
Thus, MPNNs cannot tell whether a graph is connected, as
C 2 k is but C k + C k is not.
Also, it cannot count the number of 3, 4, or 5 cycles, as C k + C k has twok cycles while C 2 k has no k cycles.
Likewise, any node in C k + C k has more length k closed walksthan any node in C 2 k .
This is because any length k closed walk in C 2 k has an analogous closed walkin C k + C k , but the nodes in C k + C k also have a closed walk that completely goes around a cycle.
I Useful Lemmas
In this section, we collect useful lemmas for our proofs.
These lemmas generally only require basictools to prove.
Our first lemma is a crucial property of quotient spaces.
Lemma 1 (Passing to the quotient) .
Let X and Y be topological spaces, and let X /G be a quotientspace, with corresponding quotient map π .
Then for every continuous G -invariant function f : X →
Y , there is a unique continuous ˜ f : X /G
→ Y such that f = ˜ f ◦ π .
Proof.
For z ∈ X /G , by surjectivity of π we can choose an x z ∈ X such that π ( x z ) =
z .
Define˜ f : X /G
→ Y by ˜ f ( z ) =
f ( x z ) .
This is well-defined, since if π ( x z ) =
π ( x ) for any other x ∈ X ,then gx z = x for some g ∈ G , so
[Equation]
where the second equality uses the G -invariance of f .
Note that ˜ f is continuous by the universalproperty of quotient spaces.
Also, ˜ f is the unique function such that f = ˜ f ◦ π ; if there were anotherfunction h : X /G
→ Y with h ( z ) ̸ =
˜ f
( z ) , then h ( z ) ̸
= f ( x z ) , so h ( π ( x z ))
=
h ( z ) ̸
= f ( x z ) .
Next, we give the First Fundamental Theorem of O ( d ) , a classical result that has been recently usedfor machine learning by Villar et al.
This result shows that an orthogonally invariant f ( V )can be expressed as a function
h ( V V ⊤ ) .
We give a proof that if f is continuous, then h is alsocontinuous.
Lemma 2 (First Fundamental Theorem of O ( d ) ) .
A continuous function f : R n × d → R s isorthogonally invariant, i.e. f ( V Q ) =
f ( V ) for all Q ∈ O ( d ) , if and only if f ( V )
=
h ( V V ⊤ ) forsome continuous h .
Proof.
If f ( V ) =
h ( V V ⊤ ) , then we have f ( V Q ) =
h ( V QQ ⊤ V ⊤ ) =
h ( V V ⊤ ) so f is orthogonallyinvariant.
For the other direction, invariant theory shows that the O ( d ) invariant polynomials are generatedby the inner products v ⊤ i v j , where v i ∈ R
d are the rows of V [Kraft and Procesi, 1996].
Let p :
R n × d → R n × n be the map p ( V ) =
V V ⊤ .
Then González and de Salas [2003] Lemma 11.13 showsthat the quotient space R n × d /O ( d ) is homeomorphic to a closed subset p ( R n × d )
=
Z ⊆ R n × n .
Let ˜ p refer to this homeomorphism, and note that ˜ p ◦ π = p by passing to the quotient (Lemma 1).
Then any continuous O ( d ) invariant f passes to a unique continuous ˜ f
: R n × d /O ( d ) → R s(Lemma 1), so f = ˜ f ◦ π where π is the quotient map.
Define h : Z → R s by h = ˜
f ◦ ˜ p − 1 , andnote that h is a composition of continuous functions and hence continuous.
Finally, we have thath ( V V ⊤ )
=
h (˜ p ◦ π ( V )) =
˜ f ◦ π ( V ) =
f ( V ) , so we are done.
The next lemma allows us to decompose a quotient of a product space into a product of smallerquotient spaces.
Lemma 3.
Let X 1 , . . .
, X k be topological spaces and G 1 , . . .
, G k be topological groups such thateach G i acts continuously on X i .
Denote the quotient maps by π i : X i → X i /G i .
Then the quotientof the product is the product of the quotient, i.e.
[Equation]
X 1 ×
[Equation] [Equation] [Equation]
X
[Equation]
G k ) , (51) and π 1 × . . .
×
π k : X 1 × . . .
X k
→ ( X 1 /G 1 ) × . . .
× ( X k /G k ) is quotient map.
Proof.
First, we show that π 1 × . . .
×
π k is a quotient map.
This is because 1.
the quotient mapof any continuous group action is an open map, so each π
i is an open map, 2.
the product of openmaps is an open map, so π 1 × . . .
×
π k is an open map and 3.
a continuous surjective open map is aquotient map, so π 1 × . . .
× π k , which is continuous and surjective, is a quotient map.
Now, we need only apply the theorem of uniqueness of quotient spaces to show (51) (see e.g. Lee[2013], Theorem A.31).
Letting q : X 1 × . . .
×
X k → ( X 1 × . . .
×
X k ) /
( G 1 × . . .
× G k ) denotethe quotient map for this space, it is easily seen that q ( x 1 , . . .
, x k ) =
q ( y 1 . . .
, y k )
if and only ifπ 1 × . . .
× π k ( x 1 , . . .
, x k ) =
π 1 × . . .
× π k
( y 1 , . . .
, y k ) , since either of these is true if and only ifthere exist
g i ∈ G i such that x i =
g
i
y
i for each i .
Thus, we have an isomorphism of these quotientspaces.
The following lemma shows that quotients of compact spaces are also compact, which is useful foruniversal approximation on quotient spaces.
Lemma 4 (Compactness of quotients of compact spaces) .
Let X be a compact space.
Then thequotient space X /G is compact.
Proof.
Denoting the quotient map by π : X → X /G and letting { U α } α be an open cover of X /G ,we have that { π − 1 ( U α ) } α is an open cover of X .
By compactness of X , we can choose a finitesubcover { π − 1 ( U α i ) } i =1 ,...,n .
Then { π ( π − 1 ( U α i ))
} i =1 ,...,n = { U α i } i =1 ,...,n by surjectivity, and 1412 { U α i } i =1 ,...,n is thus an open cover of X /G .
The Whitney embedding theorem gives a nice condition that we apply to show that the quotientspaces X /G that we deal with embed into Euclidean space.
It says that when X /G is a smoothmanifold, then it can be embedded into a Euclidean space of double the dimension of the manifold.
The proof is outside the scope of this paper.
Lemma 5 (Whitney Embedding Theorem [Whitney, 1944]) .
Every smooth manifold M of dimensionn > 0 can be smoothly embedded in R 2 n .
Finally, we give a lemma that helps prove universal approximation results.
It says that if functionsf that we want to approximate can be written as compositions f
= f L ◦ . . .
◦ f 1 , then it sufficesto universally approximate each f i and compose the results to universally approximate the f .
Thisis especially useful for proving universality of neural networks, as we may use some layers toapproximate each f i , then compose these layers to approximate the target function f .
Lemma 6 (Layer-wise universality implies universality) .
Let Z ⊆
R
d 0 be a compact domain, let
F 1 , . .
. , F L be families of continuous functions where F i consists of functions from R d i − 1 → R d ifor some d 1 , . . .
, d L .
Let F be the family of functions { f L ◦ . . .
f 1 : Z → R d L , f i ∈ F i } that arecompositions of functions f i ∈ F i .
For each i , let Φ i be a family of continuous functions that universally approximates F i .
Then thefamily of compositions
Φ = { ϕ L ◦ . . .
◦
ϕ 1 : ϕ i ∈
Φ i } universally approximates F .
Proof.
Let f = f L ◦ . . .
◦ f 1 ∈ F .
Let ˜ Z
=
Z , and then for i ≥ 2 let ˜ Z i
=
f i −
( ˜ Z i − 1 ) .
Then each˜ Z i is compact by continuity of the f i .
For 1 ≤ i < L , let Z i =
˜
Z i , and for i =
L let Z L be a compactset containing ˜ Z L such that every ball of radius one centered at a point in ˜ Z L is still contained in Z L .
Let ϵ > 0 .
We will show that there is a ϕ ∈ Φ such that ∥ f − ϕ ∥ ∞ <
ϵ by induction on L .
This holdstrivially for L = 1 , as then Φ =
Φ 1 .
Now, let L ≥ 2 , and suppose it holds for L − 1 .
By universality of Φ L , we can choose a ϕ L : Z L → 1436 R d L ∈ Φ
L such that ∥ ϕ
L − f L ∥ ∞ < ϵ/ 2 .
As ϕ L is continuous on a compact domain, it is alsouniformly continuous, so we can choose a ˜ δ > 0 such that ∥ y − z ∥ 2 < ˜ δ = ⇒ ∥
ϕ L ( y ) − ϕ L ( z ) ∥ 2 <ϵ/ 2 .
Let δ = min(˜ δ, 1) .
By induction, we can choose ϕ L − 1 ◦ . . .
◦
ϕ 1 , ϕ i ∈
Φ i such that
[Equation]
[Equation]
Note that ϕ L − 1 ◦ . . .
◦ ϕ 1 ( Z ) ⊆
Z L , because for each x ∈ Z , ϕ L − 1 ◦ . . .
◦
ϕ 1 ( x ) is within δ ≤
Euclidean distance to f L − 1 ◦ . . .
◦ f 1 ( x ) ∈ ˜
Z L , so it is contained in Z L by construction.
Thus, wemay define ϕ = ϕ
L ◦ . . .
◦
ϕ 1 : Z → R d L , and compute that
[Equation]
since ∥ ϕ L − f L ∥ ∞ < ϵ/ 2 .
To bound this other term, let x ∈ Z , and for y =
ϕ
L − 1 ◦ . . .
◦
ϕ 1 ( x )and z = f
L − 1 ◦ . . .
◦ f 1 ( x ) , we know that ∥ y − z ∥ 2 < δ , so ∥ ϕ L ( y )
− ϕ L ( z ) ∥ 2 < ϵ/ 2 by uniformcontinuity.
As this holds for all x , we have ∥ ϕ − ϕ
L ◦
f L − 1 ◦ . . .
◦
f 1 ∥ ∞ ≤ ϵ/ 2 , so ∥ ϕ − f ∥ ∞ < ϵand we are done.
[Table]
J Further Experiments
J.1 Graph Regression with no Edge Features
All graph regression models in Table 1 use edge features for learning and inference.
To show that
SignNet is also useful when no edge features are available, we ran ZINC experiments without edgefeatures as well.
The results are displayed in Table 7.
In this setting, SignNet still significantlyimproves the performance over message passing networks without positional encodings [Table] and over
Laplacian positional encodings with sign flipping data augmentation
[Table]
J.2 Learning Spectral Graph Convolutions
[Table] To numerically test the ability of our basis invariant networks for learning spectral graph convolutions,we follow the experimental setups of Balcilar et al.
He et al.
We take the dataset of 50images in He et al.
[2021] (originally from the Image Processing Toolbox of M ATLAB ) [Table] and resizethem from 100 × 100 to 32 × 32.
Then we apply the same spectral graph convolutions on them as in
He et al.
[2021], and train neural networks to learn these as regression targets.
As in prior work, wereport sum of squared errors on the training set to measure expressivity.
We compare against message passing GNNs [Kipf and Welling, 2017, Veliˇckovi´c et al., 2018] andspectral GNNs [Chien et al., 2021, Bianchi et al., 2021, Defferrard et al., 2016, He et al., 2021].
Also, we consider standard Transformers with only node features, with eigenvectors and sign flipaugmentation, and with absolute values of eigenvectors.
These models are all approximately signinvariant (they either use eigenvectors in a sign invariant way or do not use eigenvectors).
We use
DeepSets [Zaheer et al., 2017] in SignNet and 2-IGN [Maron et al., 2018] in BasisNet for ϕ , usea DeepSets for ρ in both cases, and then feed the features into another DeepSets or a standard
Transformer [Vaswani et al., 2017] to make the final predictions.
That is, we are only given graphinformation through the eigenvectors and eigenvalues, and we do not use message passing.
Table 8 displays the results, which validate our theoretical results in Section 3.1.
Without any messagepassing, SignNet and BasisNet allow DeepSets and Transformers to perform strongly, beating thespectral GNNs GPR-GNN and ARMA on all tasks.
Also, our networks outperform all other methodson the band-rejection and comb filters, and are mostly close to the best model on the other filters.
K Further Experimental DetailsK.1 Hardware, Software, and Data Details
All experiments could fit on one GPU at a time.
Most experiments were run on a server with
NVIDIA RTX 2080 Ti GPUs.
We run all of our experiments in Python, using the PyTorch [Paszkeet al., 2019] framework (license URL).
We also make use of Deep Graph Library (DGL)
[Wang et al.
(Apache License 2.0), and PyTorch Geometric (PyG)
[Fey and Lenssen, 2019] (MIT License)for experiments with graph data.
We open source our code [redacted for anonymous review].
The data we use are all freely available online.
The datasets we use are ZINC [Irwin et al., 2012],
Alchemy [Chen et al., 2019a], the synthetic counting substructures dataset [Chen et al.
, 2020],the multi-task graph property regression synthetic dataset [Corso et al., 2020] (MIT License), theimages dataset used by Balcilar et al.
[2020] (GNU General Public License v3.0), the cat mesh
fromfree3d.com/3d-model/cat-v1--522281.html (Personal Use License), and the human meshfrom turbosquid.com/3d-models/water-park-slides-3d-max/1093267 (TurboSquid 3D
Model License).
If no license is listed, this means that we cannot find a license for the dataset.
Asthey appear to be freely available with permissive licenses or no licenses, we do not ask for permissionfrom the creators or hosts of the data.
We do not believe that any of this data contains offensive content or personally identifiable information.
The 50 images used in the spectral graph convolution experiments are mostly images of objects, witha few low resolution images of humans that do not appear to have offensive content.
The only otherhuman-related data appears to be the human mesh, which appears to be from a 3D scan of a human.
The human mesh does have tattoos, but they do not appear to be offensive.
K.2 Graph Regression Details
ZINC.
In Section 4.1 we study the effectiveness of SignNet for learning positional encodings toboost the expressive power, and thereby generalization, on the graph regression problem ZINC.
Inall cases we take our ϕ encoder to be an 8 layer GIN with ReLU activation.
The input eigenvectorv i ∈ R n , where n is the number of nodes in the graph, is treated as a single scalar feature for eachnode.
In the case of using a fixed number of eigenvectors k , the aggregator ρ is taken to be an 8layer MLP with batch normalization and ReLU activation.
The aggregator ρ is applied separately tothe concatenatation of the k different embeddings for each node in a graph, resulting in one singleembedding per node.
This embedding is concatenated to the node features for that node, and the resultpassed as input to the base (predictor) model.
We also consider using all available eigenvectors ineach graph instead of a fixed number k .
Since the total number of eigenvectors is a variable quantity,equal to the number of nodes in the underlying graph, an MLP cannot be used for ρ .
To handle thevariable sized input in this case, we take ρ to be an MLP preceded by a sum over the ϕ outputs.
Inother words, the SignNet is of the form MLP (cid:16)(cid:80) ki
=1 ϕ ( v i ) + ϕ ( − v i ) (cid:17) in this case.
As well as testing SignNet, we also checked whether simple transformations that resolve the signambiguity of the Laplacian eigenvectors p
= ( v 1 , . .
. , v k ) could serve as effective positional encoding.
We considered three options.
First is to randomly flip the sign of each ± v i during training.
Thisis a common heuristic used in prior work on Laplacian positional encoding [Kreuzer et al., 2021,
Dwivedi et al., 2020].
Second, take the element-wise absolute value | v i | .
This is a non-injectivemap, creating sign invariance at the cost of destroying positional information.
Third is a differentcanonicalization that avoids stochasticity and use of absolute values by selecting the sign of eachv i so that the majority of entries are non-negative, with ties broken by comparing the ℓ 1 -norm ofpositive and negative parts.
When the tie-break also fails, the sign is chosen randomly.
Results for
GatedGCN base model on ZINC in Table 1 show that all three of these approaches are significantlypoorer positional encodings compared to SignNet.
Our training pipeline largely follows that of Dwivedi et al.
and we use the GatedGCNand PNA base models from the accompanying implementation (see https://github.com/vijaydwivedi75/gnn-lspe ).
The Sparse Transformer base model architecture we use, whichlike GAT computes attention only across neighbouring nodes, is introduced by Kreuzer et al.
Finally, the GINE implementation is based on the PyTorch Geometric implementation [Fey and
Lenssen, 2019].
For the state-of-the-art comparison, all baseline results are from their respectivepapers, except for GIN, which we run.
ZINC-full.
We also run our method on the full ZINC dataset, termed ZINC-full.
The result wereport for SignNet is a larger version of the GatedGCN base model with a SignNet that takes inall eigenvectors.
This model has 994,113 parameters in total.
All baseline results are from theirrespective papers, except for GIN, which is from [Bodnar et al., 2021].
Alchemy.
We run our method and compare with the state-of-the-art on Alchemy (with 10,000 traininggraphs).
We use the same data split as Morris et al.
[2020b].
Our base model is a GIN that takesin edge features (i.e. a GINE).
The SignNet consists of GIN for ϕ and a Transformer for ρ , as inthe counting substructures and graph property regression experiments in Section 4.2.
The modelhas 907,371 parameters in total.
Our training setting is very similar to that of Morris et al.
[2022],as we build off of their code.
We train with an Adam optimizer [Kingma and Ba, 2014] with astarting learning rate of .001, and a minimum learning rate of .000001.
The learning rate schedulecuts the learning rate in half with a patience of 20 epochs, and training ends when we reach theminimum learning rate.
All baseline results are from their respective papers, except for GIN, whichis from [Morris et al., 2022].
K.3 Spectral Graph Convolution Details
In Appendix J.2, we conduct node regression experiments for learning spectral graph convolutions.
The experimental setup is mostly taken from He et al.
However, we resize the 100 × 100images to 32 × 32 .
Thus, each image is viewed as a 1024 -node graph.
The node features X ∈ R nare the grayscale pixel intensities of each node.
Just as in He et al.
[2021], we only train andevaluate on nodes that are not connected to the boundary of the grid (that is, we only evaluate on the28 × 28 middle section).
For all experiments we limit each model to 50,000 parameters.
We use the
Adam [Kingma and Ba, 2014] optimizer for all experiments.
For each of the GNN baselines (GCN,
GAT, GPR-GNN, ARMA, ChebNet, BernNet), we select the best performing out of 4 hyperparametersettings: either 2 or 4 convolution layers, and a hidden dimension of size 32 or D , where D is justlarge enough to stay with 50,000 parameters (for instance, D = 128 for GCN, GPR-GNN, and
BernNet).
We use DeepSets or standard Transformers as our prediction network.
This takes in the output of
SignNet or BasisNet and concatenates it with the node features, then outputs a scalar prediction foreach node.
We use a 3 layer output network for DeepSets SignNet, and 2 layer output networks forall other configurations.
All networks use ReLU activations.
For SignNet, we use DeepSets for both ϕ and ρ .
Our ϕ takes in eigenvectors only, then our ρ takesthe outputs of ϕ and the eigenvalues.
We use three layers for ϕ and ρ .
For BasisNet, we use the same DeepSets for ρ as in SignNet, and 2-IGNs for the ϕ d i .
There are threedistinct multiplicities for the grid graph (1, 2, and 32), so we only need 3 separate IGNs.
Each IGNconsists of an R n 2 × 1 → R n × d ′ layer and two R n × d ′′
→
R n × d ′′′ layers, where the d ′ are hiddendimensions.
There are no matrix to matrix operations used, as the memory requirements are intensivefor these ≥ 1000 node graphs.
The ϕ d i only take in V
i V ⊤ i from the eigenspaces, and the ρ takes theoutput of the ϕ d i as well as the eigenvalues.
K.4 Substructures and Graph Properties Regression Details
We use the random graph dataset from Chen et al.
[2020] for counting substructures and the syntheticdataset from Corso et al.
[2020] for regressing graph properties.
For fair comparison we fix the basemodel as a 4-layer GIN model with hidden size 128.
We choose ϕ
as 4-layer GIN (independentlyapplied to every eigenvector) and ρ as 1-layer Transformer (independently applied to every node).
Combined with proper batching and masking, we have a SignNet that takes Laplacian eigenvectors
V ∈ R n × n and outputs fixed size sign-invariant encoding node features f ( V, Λ , X ) ∈ R n × d , where n varies between graphs but d is fixed.
We use this SignNet in our experiments and compare withother methods of handling PEs.
K.5 Texture Reconstruction Details
[Table]
We closely follow the experimental setting of Koestler et al.
[2022] for the texture reconstructionexperiments.
In this work, we use the cotangent Laplacian [Rustamov et al., 2007] of a triangle meshwith the lowest 1023 eigenvectors besides the trivial eigenvector of eigenvalue 0.
We implemented
SignNet in the authors’ original code, which was privately shared with us.
Both ρ and ϕ are takento be MLPs.
Hyperparameter settings and number of parameters are given in Table 9.
We chosehyperparameters so that the total number of parameters in the SignNet model was no larger than that
[Table]
of the original model.