S IGN AND B ASIS I NVARIANT N ETWORKS FOR S PECTRAL G RAPH R EPRESENTATION L EARNING
MIT EECS & MIT
RLE Suvrit Sra MIT LIDS Haggai
A BSTRACT
Many machine learning tasks involve processing eigenvectors derived from data.
Especially valuable are Laplacian eigenvectors, which capture useful structural information about graphs and other geometric objects.
However, ambiguities arise when computing eigenvectors: for each eigenvector v , the sign flipped − v is also an eigenvector.
More generally, higher dimensional eigenspaces contain infinitely many choices of eigenvector bases.
In this work we introduce SignNet and BasisNet — new neural architectures that are invariant to all requisite symmetries and hence process collections of eigenspaces in a principled manner.
Our networks are universal, i.e., they can approximate any continuous function of eigenvectors with the proper invariances.
They are also theoretically strong for graph representation learning — they can provably approximate any spectral graph convolution, spectral invariants that go beyond message passing neural networks, and other graph positional encodings.
Experiments show the strength of our networks for learning spectral graph filters and learning graph positional encodings.
Code is available at https://github.com/cptq/SignNet-BasisNet .
I NTRODUCTION
Numerous machine learning models process eigenvectors, which arise in various scenarios including principal component analysis, matrix factorizations, and operators associated to graphs or manifolds.
An important example is the use of graph Laplacian eigenvectors to encode information about the structure of a graph (Belkin & Niyogi, 2003).
Positional encodings that involve Laplacian eigenvectors have recently been used to generalize Transformers to graphs (Kreuzer et al., 2021; Dwivedi & Bresson, 2021), and to improve the expressive power and empirical performance of graph neural networks (GNNs) (Dwivedi et al., 2022).
Furthermore, these eigenvectors are crucial for defining spectral operations on graphs for graph signal processing and spectral GNNs (Ortega et al., 2018).
However, there are nontrivial symmetries that should be accounted for when processing eigenvectors.
If v is a unit-norm eigenvector, then so is − v , with the same eigenvalue.
More generally, if an eigenvalue has higher multiplicity, then there are infinitely many unit-norm eigenvectors that can be chosen.
Indeed, a full set of orthonormal eigenvectors is only defined up to a change of basis in each eigenspace.
In the case of sign invariance, for any k eigenvectors there are 2 k possible choices of sign.
Accordingly, prior works randomly flip eigenvector signs during training in order to approximately learn sign invariance (Kreuzer et al., 2021; Dwivedi et al., 2020).
However, learning all 2 k invariances is challenging and limits the effectiveness of Laplacian eigenvectors for encoding positional information.
Sign invariance is a special case of basis invariance when all eigenvalues are distinct, but higher dimensional basis invariance is even more difficult to deal with, and we show that these higher dimensional eigenspaces are abundant in real datasets (see Appendix B.2).
This work solves the sign and basis ambiguity problems by developing new neural networks, SignNet and BasisNet, with sign invariance and basis invariance built-in.
Our networks are universal and ∗ Equal contribution. can approximate any continuous function of eigenvectors with the proper invariances.
Moreover, our networks are theoretically powerful for graph representation learning — they can approximate spectral graph convolutions and compute graph properties like subgraph counts that message passing neural networks cannot.
Finally, Laplacian eigenvectors with SignNet and BasisNet can approximate previously proposed graph positional encodings, (Li et al., 2020; Dwivedi et al., 2022; Mialon et al., 2021; Feldman et al., 2022).
We experimentally demonstrate the strength of SignNet and
BasisNet for learning spectral graph convolutions and graph positional encodings.
S IGN AND B ASIS I NVARIANT N ETWORKS
For an n × n symmetric matrix, let λ 1 ≤ . . .
≤ λ n be eigenvalues and v 1 , . .
, v n the corresponding eigenvectors, which we may assume form an orthonormal basis.
For instance, we could consider the normalized graph Laplacian
L = I − D − 1 / 2 AD − 1 / 2 , where A ∈ R n × n is the adjacency matrix and D is the diagonal degree matrix of some undirected graph.
Our goal is to parameterize a class of models f ( v 1 , . .
, v k ) taking k eigenvectors as input that respects the eigenvector symmetries.
For instance, − v i is also an eigenvector for any v i , so a well-defined function f should be sign invariant :
[Equation]
for all sign choices s i ∈ {− 1 , 1 } .
That is, we want f to be invariant to the product group {− 1 , 1 } k .
This captures all eigenvector symmetries if the λ i are distinct.
However, if the eigenvalues have higher multiplicity, then there are further symmetries to consider.
Let V 1 , . .
, V l be bases of eigenspaces — i.e. V [Equation] [Equation]
R [Equation] i has orthonormal columns and spans the eigenspace associated with the shared eigenvalue µ i
= λ
i 1 = . . .
= λ i di .
Any other orthonormal basis that spans the eigenspace is of the form V i Q for some orthogonal Q ∈ O ( d i ) ⊆
R d i ×
d
i (see Appendix D.2).
Thus, a function f that is invariant to changes of basis in each eigenspace satisfies
[Equation]
In other words, f is invariant to the product group
O ( d 1 ) × . . .
× O
( d l ) .
The l and d i may vary between matrices, so we show below that our architectures can handle this.
Note that O (1) =
{− 1 , 1 } , so sign invariance is a special case of this basis invariance when all eigenvalues are distinct.
We often further desire such a function f to be invariant or equivariant to permutation along the entries (or rows) of each vector, as is natural for graph data.
Thus, we often require f ( PV 1 , . .
. , PV l ) =
Pf ( V 1 , . .
, V l ) for any permutation matrix P ∈ R n × n .
Figure 1 illustrates the full setup.
2. W ARMUP : N EURAL N ETWORKS ON O NE E IGENSPACE
We begin by designing sign or basis invariant neural networks on a single eigenvector or eigenspace.
For one subspace, a function h : R n → R s is sign invariant if and only if h ( v )
=
ϕ ( v ) + ϕ ( − v ) for some continuous ϕ : R n → R s , and h is permutation equivariant if
and only if ϕ is (proof in Appendix E).
We parameterize ϕ by a neural network — any architecture choice ensures sign invariance, whilst permutation equivariance can be achieved using DeepSets or most GNNs.
Switching focus to basis invariance for a single d dimensional subspace, our aim is to design maps f : R n × d → R n that are invariant to right multiplication by Q ∈ O
( d ) , and equivariant to permutations along the row axis.
We make use of the mapping V (cid:55)→ V V ⊤ , which is O ( d ) invariant.
Mapping V (cid:55)→ V V ⊤ does not lose any information if we treat V as equivalent to V Q for any
Q ∈ O ( d ) .
This is justified by the classical first fundamental theorem of O ( d ) (Kraft & Procesi, 1996), which has recently been applied in machine learning by Villar et al.
The resulting
V V ⊤ ∈ R n × n is a matrix for which permutations act on the rows and columns: we want a function φ : R n × n → R n on V V ⊤ to be permutation equivariant, so φ ( PV V ⊤ P ⊤ ) =
Pφ ( V V ⊤ ) .
To achieve this permutation equivariance from matrices to vectors, we use an invariant graph network ( IGN ) (Maron et al., 2018) — a neural network mapping to and from tensors of arbitrary order
R n d 1 → R n d 2 that has the desired permutation equivariance.
So we parameterize a family with the requisite invariance and equivariance as follows:
[Equation]
Proposition 5 shows that this architecture is universal, but this requires high order tensors to be used for IGN .
The next section discusses a more efficient architecture with restricted tensor dimensions.
2. N EURAL N ETWORKS ON M ULTIPLE E IGENSPACES
For our fully general sign invariant and basis invariant networks, eigenspaces are independently processed using the models introduced in Section 2.1, then the processed eigenspaces are aggregated into a single embedding.
This approach is motivated by the general decomposition theorem for product spaces in Section 3, which says that we may decompose along these single subspaces.
SignNet.
The unconstrained SignNet f : R n × k
→ R s on eigenvectors v 1 , . .
. , v k takes the form:
[Equation]
4) where ϕ and ρ are unrestricted neural networks, and [ · ] i denotes concatenation.
If we desire f to be a permutation equivariant function that outputs vectors in R n × s , then we restrict ϕ and ρ to be permutation equivariant networks that map vectors to vectors such as DeepSets (Zaheer et al., 2017),
Transformers (Vaswani et al., 2017), or GNNs.
We call the permutation equivariant version SignNet .
BasisNet.
We take a similar approach for basis invariance.
Let V i ∈ R n × d
i be an orthonormal basis of a d i dimensional eigenspace.
Then we parameterize our unconstrained BasisNet f by
[Equation]
where each ϕ d i is shared amongst all subspaces of the same dimension d i , and l is the number of eigenspaces.
As l differs between different graphs, we may use zero-padding or a sequence model like a Transformer to parameterize ρ .
Again, ϕ d i and ρ are generally unrestricted neural networks.
If we want permutation equivariance, then we let ρ be permutation equivariant and ϕ d i =
IGN
d
i :
R n 2 → R n be IGNs from matrices to vectors.
For efficiency, we will only use matrices and vectors in the IGNs (that is, no tensors in R n p for p > 2 ), i.e., we use 2-IGN.
Our resulting BasisNet is
[Equation]
U NIVERSALITY AND T HEORETICAL E XPRESSIVE P OWER
Without permutation equivariance constraints, the universality of both architectures introduced in
Section 2.2 follow as corollaries of the following general decomposition result.
Theorem 1 (Decomposition Theorem) .
Let X 1 , . . .
, X k be topological spaces, and let G i be a group acting on X i for each i .
We assume mild topological conditions on X i and G i hold.
For any continuous f : X = X 1 × . . .
×
X k
→ R s that is invariant to the action of G =
G 1 × . . .
× G k , there exists continuous ϕ
i and a continuous ρ such that
[Equation]
Furthermore: (1) each ϕ i can be taken to be invariant to G i , (2) the domain Z of ρ is compact if each X i is compact, (3) if X i =
X j
and G i =
G j , then ϕ i can be taken to be equal to ϕ j .
This result says that when a product of groups G acts on a product of spaces X , for invariance to the product group G it suffices to individually process each smaller group G i on X
i
and then aggregate the results.
Along with the proof of Theorem 1, the mild topological assumptions are explained in
Appendix E.1.
The assumptions hold for sign invariance and basis invariance.
[Table]
[Table]
Corollary 1.
Unconstrained SignNet can represent any sign invariant function and unconstrained
BasisNet can represent any basis invariant function. Now
[Table]
we prove that our SignNets and BasisNets can compute useful basis invariant and permutation equivariant functions on Laplacian eigenvectors for graph representation learning.
SignNets and BasisNets Generalize Spectral Graph Convolution For node features X ∈ R n × q and an eigendecomposition V Λ V ⊤ , a spectral graph convolution takes the form f ( V, Λ , X )
= (cid:80) ni
=
θ i v i v ⊤ i X , for parameters
θ i , that may optionally be continuous functions
h ( λ i )
=
θ i of the eigenvalues (Bruna et al., 2014; Defferrard et al., 2016).
This includes important functions like heat kernels and generalized PageRanks on graphs (Li et al., 2019).
Spectral graph convolutions are permutation equivariant and sign invariant, and if θ i
= h ( λ i ) , they are invariant to change of bases in each eigenspace.
Our SignNet and BasisNet universally approximate spectral graph convolutions.
See Appendix F.1 for proofs.
Proposition 1.
SignNet universally approximates all spectral graph convolutions.
BasisNet universally approximates all parametric spectral graph convolutions.
BasisNets can Compute Spectral Invariants
Many works compare the expressive power of
GNNs to combinatorial invariants on graphs, especially the k -Weisfeiler-Lehman ( k -WL) tests of graph isomorphism.
We argue that it is more natural to analyze our methods in terms of spectral invariants , which are computed from the eigenvalues and eigenvectors of graphs.
A widely studied spectral invariant is the collection of graph angles, which are defined as the values α ij =
∥ V i V ⊤ i e j ∥ 2 , where V i ∈ R n × d
i is an orthonormal basis for the i th adjacency matrix eigenspace, and e j is the j th standard basis vector.
These are easily computed by our networks, and thus our networks inherit the strength of these invariants for learning functions on graphs.
Proposition 2.
BasisNet universally approximates the graph angles α ij .
The eigenvalues and graph angles (and thus BasisNets) can determine the number of length 3, 4, or 5 cycles, whether a graph is connected, and the number of length k closed walks from any vertex to itself. See Appendix F.3 for proofs.
In contrast, message passing GNNs are not able express any of these properties (Arvind et al., 2020; Chen et al., 2020).
Although spectral invariants are strong, F¨urer (2010) shows that the eigenvalues and graph angles are not stronger than the 3-WL test (i.e. the 2-FWL test).
In Appendix H, we run experiments on counting substructures in graphs with SignNet.
The results show that SignNet outperforms message passing GNNs and Laplacian PEs with sign flipping data augmentation for counting these substructures.
SignNet and BasisNet can express existing positional encodings There have been many graph positional encodings proposed in the literature, and it is unclear when one would choose one over the other.
We show that SignNet and BasisNet can approximate many previously used graph positional encodings, as captured in the following proposition.
This is because these positional encodings can be expressed as part of a spectral graph convolution matrix.
See Appendix F.2 for proofs. Proposition 3. SignNet and BasisNet universally approximate heat kernel positional encodings (Feldman et al., 2022) and random walk node positional encodings (RWPE) (Dwivedi et al., 2022). BasisNet universally approximates diffusion and p -step random walk relative positional encodings (Mialon et al., 2021), as well as generalized PageRank and landing probability distance encodings
(Li et al., 2020).
E XPERIMENTS
Graph Regression.
We study the effectiveness of SignNet in learning positional encodings (PEs) from Laplacian eigenvectors on the ZINC dataset.
We primarily consider three settings: 1) No positional encoding, 2) Laplacian PE (LapPE) – the k eigenvectors of the graph Laplacian with smallest eigenvalue are concatenated with existing node features, 3) SignNet positional features – passing the eigenvectors through a SignNet and concatenating the output with node features.
We also consider ablations in which we take the absolute values of the eigenvectors, choose a canonical sign for each eigenvector by maximizing the norm of positive entries, or use a variant of our SignNet where we process each eigenspace by ϕ ( v ) instead of ϕ ( v ) + ϕ ( − v ) .
See Appendix I.2 for details.
Results comparing different positional encoding methods are displayed in Table 1.
For all 4 base model, the PE learned with SignNet yields the best test MAE (lower is better).
Notably, this includes the cases of PNA and GINE, for which Laplacian PE with simple random sign flipping was unable to improve performance over using no PE at all.
Learning Spectral Graph Convolutions.
We test the ability of our basis invariant networks for learning spectral graph convolutions.
We compare against spectral GNNs (Bianchi et al., 2021;
Defferrard et al., 2016; He et al., 2021).
Also, we consider standard Transformers with only node features, with eigenvectors and sign flip augmentation, and with absolute values of eigenvectors.
These models all either use eigenvectors in a sign invariant way or do not use eigenvectors.
As in prior work (He et al., 2021; Balcilar et al., 2020), we measure performance on the whole training set, so that we can measure expressivity of the models.
We use DeepSets in SignNet and 2-IGN (Maron et al., 2018) in BasisNet for ϕ , use a DeepSets for ρ in both cases, and then feed the features into another DeepSets or a standard Transformer (Vaswani et al., 2017) to make the final predictions.
That is, we are only given graph information through the eigenvectors and eigenvalues, and we do not use message passing.
Table 2 displays the results.
SignNet and BasisNet allow DeepSets and Transformers to outperform all other methods on the band-rejection and comb filters, and they are mostly close to the best model on the other filters. A CKNOWLEDGMENTS
Stefanie Jegelka and Suvrit Sra acknowledge support from NSF CCF-2112665 (TILOS AI Research
Institute).
Stefanie Jegelka also acknowledges support from NSF Award 2134108 and NSF Convergence Accelerator Track D 2040636 and NSF C-ACCEL D636 - CRIPT Phase 2. Suvrit Sra acknowledges support from NSF CAREER grant (IIS-1846088). Joshua Robinson is partially supported by a Two Sigma fellowship. A. G RAPH P OSITIONAL E NCODINGS
Various graph positional encodings have been proposed, which have been motivated for increasing expressive power or practical performance of graph neural networks, and for generalizing Transformers to graphs.
Positional encodings are related to so-called position-aware network embeddings (Chami et al., 2020), which capture distances between nodes in graphs.
These include network embedding methods like Deepwalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016), which have been recently integrated into GNNs that respect their invariances by Wang et al.
Further, Li et al.
(2020) studies the theoretical and practical benefits of incorporating distance features into graph neural networks.
Dwivedi et al.
(2022) proposes a method to inject learnable positional encodings into each layer of a graph neural network, and uses a simple random walk based node positional encoding.
You et al.
(2021) proposes a node positional encoding diag( A k ) , which captures the number of closed walks from a node to itself.
Dwivedi et al.
(2020) propose to use Laplacian eigenvectors as positional encodings in graph neural networks, with sign ambiguities alleviated by sign flipping data augmentation.
While positional encodings in sequences as used for Transformers (Vaswani et al., 2017) are able to leverage the canonical order in sequences, there is no such useful canonical order for nodes in a graph, due in part to permutation symmetries.
Thus, different permutation equivariant positional encodings have been proposed to help generalize Transformers to graphs.
Dwivedi & Bresson (2021) directly adds in linearly projected Laplacian eigenvectors to node features before processing these features with a Transformer.
Kreuzer et al.
(2021) propose an architecture that uses attention over
Laplacian eigenvectors and eigenvalues to learn node or edge positional encodings.
Mialon et al.
(2021) uses spectral kernels such as the diffusion kernel to define relative positional encodings that modulate the attention matrix.
Ying et al.
(2021) achieve state-of-the-art empirical performance with simple Transformers that incorporate shortest-path based relative positional encodings.
Zhang et al.
(2020) also utilizes shortest-path distances for positional encodings in their graph Transformer.
A. E IGENVECTOR S YMMETRIES IN G RAPH R EPRESENTATION L EARNING
Many works that attempt to respect the invariances of eigenvectors solely focus on sign invariance (Dwivedi et al., 2020; Dwivedi & Bresson, 2021; Dwivedi et al., 2022; Kreuzer et al., 2021).
This may be reasonable for continuous data, where eigenvalues of associated matrices may be usually distinct and separated (e.g. Puny et al.
(2022) finds that this empirically holds for covariance matrices of n -body problems).
However, discrete graph Laplacians are known to have higher multiplicity eigenvalues in many cases, and in Appendix B.2 we find this to be true in various types of real-world graph data.
Graphs without higher multiplicity eigenspaces are easier to deal with; in fact, graph isomorphism can be tested in polynomial time on graphs of bounded multiplicity for
A recent work of Wang et al.
(2022) proposes full orthogonal group invariance for functions that process positional encodings.
In particular, for positional encodings Z ∈ R n × k , they parameterize functions f ( Z ) such that f ( Z ) =
f ( ZQ ) for all Q ∈ O ( k ) .
This indeed makes sense for network embeddings like node2vec (Grover & Leskovec, 2016), as their objective functions are based on inner products and are thus orthogonally invariant.
While they prove stability results when enforcing full orthogonal invariance for eigenvectors, this is too strict of a constraint compared to our basis invariance.
For instance, when k = n and all eigenvectors are used in V , the condition f ( V ) =
f ( V Q ) implies that f is a constant function on orthogonal matrices, since any orthogonal matrix W can be obtained as W = V Q for Q =
V ⊤ W ∈ O ( n ) .
In other words, for bases of eigenspaces V 1 , . . .
, V l and V =
[ V. . .
V l ] , Wang et al.
(2022) enforces V Q
∼ =
V , while we enforce V Diag( Q 1 , . .
. , Q l ) ∼ = V .
While the columns of V Diag( Q 1 , . . . , Q l ) are still eigenvectors, the columns of V Q generally are not.
A. G RAPH S PECTRA AND L EARNING ON G RAPHS
More generally, graph spectra is widely used in analyzing graphs, and spectral graph theory (Chung, 1997) studies the connection between graph properties and graph spectra.
Different graph kernels have been defined based on graph spectra, which use robust and discriminative notions of generalized spectral distance (Verma & Zhang, 2017), the spectral density of states (Huang et al., 2021), random walk return probabilities (Zhang et al., 2018), or the trace of the heat kernel (Tsitsulin et al., 2018).
Graph signal processing relies on spectral operations to define Fourier transforms, frequencies, convolutions, and other useful concepts for processing data on graphs (Ortega et al., 2018).
The closely related spectral graph neural networks (Wu et al., 2020; Balcilar et al., 2020) parameterize neural architectures that are based on similar spectral operations.
B E IGENVECTORS AND E IGENVALUES IN P RACTICE
In this section, we study the properties of eigenvalues and eigenvectors computed by numerical algorithms on real-world data. B. S IGN AND B ASIS A MBIGUITIES IN N UMERICAL E IGENSOLVERS When learning from real-world data, we use eigenvectors that are computed by numerical algorithms.
These algorithms return specific eigenvectors for each eigenspace, so there is some theoretically arbitrary choice of sign or basis of each eigenspace.
The general symmetric matrix eigensolvers numpy.linalg.eigh and scipy.linalg.eigh both call LAPACK routines.
They both proceed as follows: for a symmetric matrix A , they first decompose it as A =
QTQ ⊤ for orthogonal Q and tridiagonal T , then they compute the eigendecomposition of T = W Λ W ⊤ , so the eigendecomposition of A is A = ( QW )Λ( W ⊤ Q ⊤ ) .
There are multiple ambiguities here: for diagonal sign matrices S
=
Diag( s 1 , . . .
, s n ) and S ′ =
Diag( s ′ 1 , . . .
, s ′ n ) , where s i , s ′ i ∈ {− 1 , 1 } , we have that A = QS ( STS ) SQ ⊤ is also a valid tridiagonalization, as QS is still orthogonal,
SS = I , and STS is still tridiagonal.
Also, T = ( WS ′ )
Λ( S ′ W ⊤ ) is a valid eigendecomposition of T , as
WS ′ is still orthogonal.
In practice, the general symmetric matrix eigensolvers numpy.linalg.eigh and scipy.linalg.eigh differ between frameworks but are consistent with the same framework.
More specifically, for a symmetric matrix A , we find that the eigenvectors computed with the default settings in numpy tend to differ by a choice of sign or basis from those that are compute with the default settings in scipy.
On the other hand, the called LAPACK routines are deterministic, so the eigenvectors returned by numpy are the same in each call, and the eigenvectors returned by scipy are likewise the same in each call.
Eigensolvers for sparse symmetric matrices like scipy.linalg.eigsh are required for large scale problems.
This function calls ARPACK, which uses an iterative method that starts with a randomly sampled initial vector.
Due to this stochasticity, the sign and basis of eigenvectors returned differs between each call.
[Table]
Bro et al.
(2008) develops a data-dependent method to choose signs for each singular vector of a singular value decomposition.
Still, in the worst case the signs chosen will be arbitrary, and they do not handle rotational ambiguities in higher dimensional eigenspaces.
Other works have made choices of sign, such as by picking the sign so that the eigenvector’s entries are in the largest lexicographic order (Tam & Dunson, 2022).
This choice of sign may work poorly for learning on graphs, as it is sensitive to permutations on nodes.
For some graph regression experiments in
Section 4, we try a choice of sign that is permutation invariant, but we find it to work poorly.
B. H IGHER D IMENSIONAL E IGENSPACES IN R EAL G RAPHS
Here, we investigate the normalized Laplacian eigenspace statistics of real world graph data.
For any graph that has distinct Laplacian eigenvalues, only sign invariance is required in processing eigenvectors.
However, we find that graph data tends to have higher multiplicity eigenvalues, so basis invariance would be required for learning symmetry-respecting functions on eigenvectors.
Indeed, we show statistics for multi-graph datasets in Table 3 and for single-graph datasets with more nodes per graph in Table 4.
For multi-graph datasets, we consider :
• Molecule graphs: ZINC (Irwin et al., 2012; Dwivedi et al., 2020), ogbg-molhiv (Wu et al., 2018; Hu et al., 2020a)
• Social networks: IMDB-M, COLLAB (Yanardag & Vishwanathan, 2015; Morris et al., 2020), • Bioinformatics graphs: PROTEINS (Morris et al., 2020) • Computer vision graphs: COIL-DEL (Riesen & Bunke, 2008; Morris et al., 2020).
For single-graph datasets, we consider:
• The 32 × 32 image grid as in Section• Citation networks: Cora, Citeseer (Sen et al., 2008)
• Co-purchasing graphs with Amazon Photo (McAuley et al., 2015; Shchur et al., 2018).
We see that these datasets all contain higher multiplicity eigenspaces, so sign invariance is insufficient for fully respecting symmetries.
The majority of graphs in each multi-graph dataset besides
COIL-DEL contain higher multiplicity eigenspaces.
Also, the dimension of these eigenspaces can be quite large compared to the size of the graphs in the dataset.
The single-graph datasets have a large proportion of their eigenvectors belonging to higher dimensional eigenspaces.
Thus, basis invariance may play a large role in processing spectral information from these graph datasets.
C V ISUALIZATION OF L EARNED P OSITIONAL E NCODINGS
To better understand SignNet, in Figure 3 we visualize the learned positional encodings of a SignNet with ϕ =
GIN, ρ = MLP trained on ZINC as in Section 4.
In the top row, we plot the normalized Laplacian eigenvectors of lowest and highest frequencies (besides the trivial eigenvector corresponding to the zero eigenvalue) for the Fluorescein molecule (we arbitrarily chose a molecule
[Table]
with several rings from the internet).
In the bottom row, we plot the first four principal components of the n × s dimensional output of our SignNet ρ ([ ϕ ( v i ) + ϕ ( − v i )] i =1 ,...,n ) on all nontrivial eigenvectors.
SignNet learns interesting structural information such as min-cuts (PC 3) and appendage atoms (PC 2) that qualitatively differ from any single eigenvector of the graph.
D D EFINITIONS , N OTATIONS , AND B ACKGROUND D. B ASIC T OPOLOGY AND A LGEBRA D EFINITIONS
We will use some basic topology and algebra for our theoretical results.
A topological space ( X , τ ) is a set X along with a family of subsets τ ⊆ 2 X satisfying certain properties, which gives useful notions like continuity and compactness.
From now on, we will omit mention of τ , and refer to a topological space as the set X itself.
For topological spaces X and Y , we write X ∼ =
Y and say that
X is homeomorphic to Y if there exists a continuous bijection with continuous inverse from X to
Y .
We will say X =
Y if the underlying sets and topologies are equal as sets (we will often use this notion of equality for simplicity, even though it can generally be substituted with homeomorphism).
For a function f : X → Y between topological spaces X and Y , the image im f is the set of values that f takes, im f
= { f ( x ) : x ∈ X} .
This is also denoted f ( X ) .
A function f : X → Y is called a topological embedding if it is a homeomorphism from X to its image.
A group G is a set along with a multiplication operation G × G → G , such that multiplication is associative, there is a multiplicative identity e ∈ G , and each g ∈ G has a multiplicative inverse g − 1 .
A topological group is a group that is also a topological space such that the multiplication and inverse operations are continuous.
A group G may act on a set X by a function · : G × X → X .
We usually denote g · x as gx .
A topological group is said to act continuously on a topological space X if · is continuous.
For any group G and topological space X , we define the coset
Gx = { gx : x ∈ X} , which can be viewed as an equivalance class of elements that can be transformed from one to another by a group element.
The quotient space X /G
=
{ Gx : x ∈ X} is the set of all such equivalence classes, with a topology induced by X .
For x ∈ R s , ∥ x ∥ 2 denotes the standard Euclidean norm.
By the ∞ norm of functions f : Z → R s from a compact Z to a Euclidean space R s , we mean ∥ f ∥ ∞ = sup z ∈Z ∥
f ( z ) ∥ 2 .
D.
B ACKGROUND ON E IGENSPACE I NVARIANCES
Let V =
[ v. . .
v d ] and W = [ w. . .
w d ] ∈ R n × d be two orthonormal bases for the same d dimensional subspace of R n .
Since V and W span the same space, their orthogonal projectors are the same, so V V ⊤ = WW ⊤ .
Also, since V and W have orthonormal columns, we have
V ⊤ V = W ⊤ W = I ∈ R d × d . Define Q = V ⊤ W . Then Q is orthogonal because
[Equation]
Moreover, we have that
[Equation]
Thus, for any orthonormal bases V and W of the same subspace, there exists an orthogonal Q ∈ O ( d ) such that V Q = W .
For another perspective on this, define the Grassmannian Gr( d, n ) as the smooth manifold consisting of all d dimensional subspaces of R n .
Further define the Stiefel manifold St( d, n ) as the set of all orthonormal tuples [ v. . .
v d ] ∈ R n × d of d vectors in R n .
Letting O ( d ) act by right multiplication, it holds that St( d, n ) /O
( d ) ∼ =
Gr( d, n ) .
This implies that any O ( d ) invariant function on
St( d, n ) can be viewed as a function on subspaces.
See e.g. Gallier & Quaintance (2020) Chapterfor more information on this.
We will use this relationship in our proofs of universal representation.
When we consider permutation invariance or equivariance, the permutation acts on dimensions of size n .
Then a tensor X ∈ R n k × d is called an order k tensor with respect to this permutation symmetry, where order 0 are called scalars, order 1 tensors are called vectors, and order 2 tensors are called matrices.
Note that this does not depend on d ; in this work, we only ever consider vectors and scalars with respect to the O ( d ) action.
E P ROOFS OF U NIVERSALITY
We begin by proving the two propositions for the single subspace case from Section 2.1.
Proposition 4.
A continuous function h : R n → R s is sign invariant if and only if
[Equation]
for some continuous ϕ : R n → R s .
A continuous h : R n → R n is sign invariant and permutation equivariant if and only if equation 10 holds for a continuous permutation equivariant ϕ : R n → R n .
Proof.
If h ( v ) =
ϕ ( v ) + ϕ ( − v ) , then h is obviously sign invariant.
On the other hand, if h is sign invariant, then letting ϕ ( v ) =
h ( v ) / 2 gives that h ( v ) =
ϕ ( v ) + ϕ ( − v ) , and ϕ is of course continuous.
If h ( v ) =
ϕ ( v ) + ϕ ( − v ) for a permutation equivariant ϕ , then h ( − Pv ) =
ϕ ( − Pv ) + ϕ ( Pv ) =
Pϕ ( − v )+ Pϕ ( v ) =
P ( ϕ ( v )+ ϕ ( − v ))
=
Ph ( v ) , so h is permutation equivariant and sign invariant.
If h is permutation equivariant and sign invariant, then define ϕ ( v )
=
h ( v ) / 2 again; it is clear that ϕ is continuous and permutation equivariant.
Proposition 5.
Any continuous, O ( d ) invariant h : R n × d → R s is of the form h ( V ) =
φ ( V V ⊤ ) for a continuous φ .
For a compact domain Z ⊆ R n × d , maps of the form V (cid:55)→ IGN( V V ⊤ ) universally approximate continuous functions
h : Z ⊆ R n × d → R n that are O ( d ) invariant and permutation equivariant.
Proof.
The case without permutation equivariance holds by the First Fundamental Theorem of O ( d ) (Lemma 2).
For the permutation equivariant case, let Z ′
=
{ V V ⊤ : V ∈ Z} and let ϵ > 0 .
Note that Z ′ is compact, as it is the continuous image of a compact set.
Since h is O ( d ) invariant, the first fundamental theorem of O ( d ) shows that there exists a continuous function φ : Z ′ ⊆
R n × n → R n such that h ( V ) =
φ ( V V ⊤ ) .
Since h is permutation equivariant, for any permutation matrix P we have that
[Equation]
so φ is a continuous permutation equivariant function from matrices to vectors.
Then note that
Keriven & Peyr´e (2019) show that invariant graph networks (of generally high tensor order in hidden layers) universally approximate continuous permutation equivariant functions from matrices to vectors on compact sets of matrices.
Thus, an IGN can ϵ -approximate φ , and hence V (cid:55)→ IGN( V V ⊤ ) can ϵ -approximate h .
E. P ROOF OF D ECOMPOSITION T HEOREM
[Equation]
Here, we give the formal statement of Theorem 1, which provides the necessary topological assumptions for the theorem to hold.
In particular, we only require the G i be a topological group that acts continuously on X i for each i , and that there exists a topological embedding of each quotient space into some Euclidean space.
The first assumption is very mild, and it holds for any finite or compact matrix group, which all of the invariances we consider in this paper can be represented as.
Also, many different conditions can guarantee existence of a topological embedding of the quotient space in a Euclidean space.
For instance, if the quotient space is a smooth manifold, then the Whitney
Embedding Theorem (Lemma 5) guarantees such an embedding.
Also, if the base space X i is a
Euclidean space and G i is a finite or compact matrix Lie group, then a map built from G -invariant polynomials gives such an embedding (Gonz´alez & de Salas (2003) Lemma 11.13).
Figure 4 provides a commutative diagram representing the constructions in our proof.
Theorem 1 (Decomposition Theorem) .
Let X 1 , . . .
, X k be topological spaces, and let G i be a topological group acting continuously on X i for each i .
Assume that there is a topological embedding ψ
i : X i /G i
→ R a i of each quotient space into a Euclidean space R a i for some dimension a i .
Then, for any continuous function f : X = X 1 × . . .
×
X k
→ R s that is invariant to the action of
G = G 1 × . . .
× G k , there exists continuous functions ϕ
i : X i → R a i and a continuous function ρ : Z ⊆ R a → R s , where a = (cid:80) i a i such that
[Equation]
Furthermore: (1) each ϕ i can be taken to be invariant to G i , (2) the domain Z is compact if each
X i is compact, (3) if X i =
X j
and G i =
G j , then ϕ i can be taken to be equal to ϕ j .
Proof.
Let π i :
X i → X i /G
i denote the quotient map for X i /G i .
Since each G i acts continuously,
Lemma 3 gives that the quotient of the product space is the product of the quotient spaces, i.e. that
[Equation]
) and the corresponding quotient map π : X /G is given by
[Equation]
By passing to the quotient (Lemma 1), there exists a continuous ˜ f : X / G → R s on the quotient space such that f = ˜ f ◦ π .
By Lemma 4, each X i /G
i is compact if X i is compact.
Defining the image Z i =
ψ i ( X i /G i ) ⊆
R a i , we thus know that Z i is compact if X i is compact.
Moreover, as ψ i is a topological embedding, it has a continuous inverse ψ − 1 i on its image Z i .
Further, we have a topological embedding ψ : X /G
→ Z = Z 1 × . . .
×Z k given by ψ = ψ 1 × . . .
× ψ k , with continuous inverse ψ − 1 = ψ − 1 1 × . . .
× ψ − 1 k .
Note that
[Equation]
[Equation]
So we define
[Equation]
Thus, f
= ρ ◦ ϕ
=
ρ ◦ ( ϕ 1 × . . .
× ϕ k ) , so equation equation 7 holds.
Moreover, the ρ and ϕ i are continuous, as they are compositions of continuous functions.
Furthermore, (1) holds as each ϕ
i is invariant to G i because each π i is invariant to G i .
Since each Z i is compact if X i is compact, the product Z
= Z 1 × . . .
× Z k is compact if each X i is compact, thus proving (2).
To show the last statement (3), note simply that if X i =
X j
and G i =
G j , then the quotient maps are equal, i.e. π i = π j .
Moreover, we can choose the embeddings to be equal, so say ψ i = ψ j .
Then, ϕ i =
ψ i ◦ π i =
ψ j ◦ π j = ϕ j , so we are done.
E. U NIVERSALITY OF S IGN N ET AND B ASIS N ET
Here, we prove Corollary 1 on the universal representation and approximation capabilities of our unconstrained SignNets and unconstrained BasisNets.
We proceed in several steps.
Expressive-BasisNet.
While we restrict SignNet to only process vectors and BasisNet to only process vectors and matrices, higher order tensors are generally required for universality in expressing permutation equivariant or invariant networks (Keriven & Peyr ´ e, 2019; Maron et al., 2019).
Thus, we will consider a theoretically powerful but computationally impractical variant of our model, in which we replace ρ and IGN
d i in BasisNet with IGNs of arbitrary tensor order.
We call this variant
Expressive-BasisNet .
In this Appendix, we will also prove that Expressive-BasisNet can universally approximate basis invariant and permutation equivariant continuous functions.
E.2. S IGN I NVARIANT U NIVERSAL R EPRESENTATION
Recall that S n − 1 denotes the unit sphere in R n .
As we normalize eigenvectors to unit norm, the domain of our functions on k eigenvectors are on the compact space ( S n − 1 ) k .
Corollary 2 (Universal Representation for SignNet) .
A continuous function f : ( S n − 1 ) k
→ R s is sign invariant, i.e. f ( s 1 v 1 , . . .
, s k v k ) =
f ( v 1 , . .
, v k ) for any s i ∈ {− 1 , 1 } , if and only if there exists a continuous ϕ : R n → R 2 n − 2 and a continuous ρ : R (2 n − 2) k
→ R s such that
[Equation]
Proof.
It can be directly seen that any f of the above form is sign invariant.
Thus, we show that any sign invariant f can be expressed in the above form.
First, we show that we can apply the general Theorem 1.
The group G i = { 1 , − 1 } acts continuously and satisfies that S n − 1 / { 1 , − 1 } = RP n − 1 , where RP n − 1 is the real projective space of dimension n − 1 .
Since
RP n − 1 is a smooth manifold of dimension n − 1 , Whitney’s embedding theorem states that there exists a (smooth) topological embedding ψ i : RP n − 1 → R 2 n − 2 (Lemma 5).
Thus, we can apply the general theorem to see that f = ρ ◦ ˜ ϕ
k for some continuous ρ and ˜ ϕ k .
Note that each ˜ ϕ
i =
˜ ϕ is the same, as each X i =
S n − 1 and G i = { 1 , − 1 } is the same.
Also, Theoremsays that we may assume that ˜ ϕ is sign invariant, so ˜ ϕ
( x ) =
˜ ϕ
( − x ) .
Letting ϕ
( x ) =
˜ ϕ
( x ) / 2 , we are done with the proof.
E.2. S IGN I NVARIANT U NIVERSAL R EPRESENTATION WITH E XTRA F EATURES
Recall that we may want our sign invariant functions to process other data besides eigenvectors, such as eigenvalues or node features associated to a graph.
Here, we show universal representation for when we have this other data that does not possess sign symmetry.
The proof is a simple extension of Corollary 2, but we provide the technical details for completeness.
Corollary 3 (Universal Representation for SignNet with features) .
For a compact space of features Ω ⊆ R d , let f ( v 1 , . .
. , v k , x 1 , . . .
, x k ) be a continuous function f : ( S n − 1 × Ω) k
→ R s .
Then f is sign invariant for the inputs on the sphere, i.e.
[Equation]
if and only if there exists a continuous ψ : R n + d → R 2 n − 2+ d and a continuous ρ : R (2 n − 2+ d ) k
→
R s such that
[Equation]
Proof.
Once again, the sign invariance of any f in the above form is clear.
We follow very similar steps to the proof of Corollary 2 to show that we may apply Theorem 1.
We can view Ω as a quotient space, after quotienting by the trivial group that does nothing, Ω ∼ = Ω / { 1 } .
The corresponding quotient map is id Ω , the identity map.
Also, Ω trivially topologically embeds in R d by the inclusion map.
As G i = {− 1 , 1 } × { 1 } acts continuously, by Lemma 3 we have that
[Equation]
with corresponding quotient map π × id Ω , where π is the quotient map to RP n − 1 .
Letting ˜ ψ be the embedding of RP n − 1 → R 2 n − 2 guaranteed by Whitney’s embedding theorem (Lemma 5), we have that ψ = ˜
ψ
× id Ω is an embedding of RP n − 1 × Ω
→ R 2 n − 2+ d .
Thus, we can apply Theorem 1 to write f = ρ ◦ ˜ ϕ k for ˜ ϕ
= ( ˜ ψ × id Ω ) ◦ ( π × id Ω ) , so
[Equation]
[Equation]
(24) where ˜ ϕ ( v i , x i ) =
˜ ϕ
( − v i , x i ) .
Letting ϕ
( v i , x i ) =
˜ ϕ
( v i , x i ) / 2 , we are done.
E.2. B ASIS I NVARIANT U NIVERSAL R EPRESENTATION
Recall that St( d, n ) is the Stiefel manifold of k -tuples of vectors ( v 1 , . .
, v k ) where v i ∈ R n and v 1 , . .
. , v k are orthonormal.
This is where our inputs lie, as our eigenvectors are unit norm and orthogonal.
We will also make use of the Grassmannian Gr( d, n ) , which consists of all d -dimensional subspaces in R n .
This is because the Grassmannian is the quotient space for the group action we want, Gr( d, n ) ∼ =
St( d, n ) /O
( d ) , where Q ∈ O ( d ) acts on V ∈ St( d, n ) ⊆
R n × d by mapping V to
V Q (Gallier & Quaintance, 2020).
Corollary 4 (Universal Representation for BasisNet) .
For dimensions d 1 , . . .
, d l ≤ n let f be a continuous function on St( d 1 , n ) × . . .
×
St( d l , n ) .
Further assume that f is invariant to O ( d 1 ) × . . .
× O
( d l ) , where O ( d i ) acts on St( d i , n ) by multiplication on the right.
Then there exist continuous ρ : R (cid:80) li =1 2 d i ( n − d i )
→ R s and continuous ϕ
i : St( d i , n ) → R 2 d i ( n − d i ) such that
[Equation]
where the ϕ i are O
( d i ) invariant functions, and we can take ϕ i = ϕ
j if d i =
d j .
Proof.
Letting X i
=
St( d i , n ) and G i
= O ( d i ) , it can be seen that G i acts continuously on X i .
Also, we have that the quotient space St( d i , n ) /O
( d i ) =
Gr( d i , n ) is the Grassmannian of d i dimensional subspaces in R n , which is a smooth manifold of dimension
d i ( n − d i ) .
Thus, the Whitney embedding theorem (Lemma 5) gives a topological embedding ψ
i : Gr( d i , n ) → R 2 d i ( n − d i ) .
Hence, we may apply Theorem 1 to obtain continuous O
( d i )
invariant ϕ
i : St( d i , n ) → R 2 d i ( n − d i ) and continuous ρ : R (cid:80) li =
1 2 d i ( n − d i )
→ R s , such that f = ρ ◦ ( ϕ 1 × . . .
× ϕ l ) .
Also, if d i =
d j , then
X i =
X j
and G i =
G j , so we can take ϕ i = ϕ j .
E.2. B ASIS I NVARIANT AND P ERMUTATION E QUIVARIANT U NIVERSAL A PPROXIMATION
With the restriction that f ( V 1 , . . .
, V l ) : R n × (cid:80) i d i → R n be permutation equivariant and basis invariant, we need to use the impractically expensive Expressive-BasisNet to approximate f .
Universality of permutation invariant or equivariant functions from matrices to scalars or matrices to vectors is difficult to achieve in a computationally tractable manner (Maron et al., 2019;
Keriven &
Peyr´e, 2019).
One intuitive reason to expect this is that universally approximating such functions allows solution of the graph isomorphism problem (Chen et al., 2019), which is a computationally difficult problem.
While we have exact representation of basis invariant functions by continuous ρ and ϕ i when there is no permutation equivariance constraint, we can only achieve approximation up to an arbitrary ϵ > 0 when we require permutation equivariance.
Corollary 5 (Universal Approximation for Expressive-BasisNets) .
Let f ( V 1 , . .
, V l ) :
St( d 1 , n ) × . . .
×
St( d l , n ) → R n be continuous, O ( d 1 ) × . . .
× O ( d l ) invariant, and permutation equivariant.
Then f can be ϵ -approximated by an Expressive-BasisNet.
Proof.
By invariance, Corollary 4 of the decomposition theorem shows that f can be written as
[Equation]
26) for some continuous O ( d i ) invariant ϕ d i and continuous ρ .
By the first fundamental theorem of
O ( d ) (Lemma 2), each ϕ d i can be written as ϕ d
i ( V i ) =
φ d i ( V i , V ⊤ i ) for some continuous φ d i .
Let
[Equation]
27) which is compact as it is the image of the compact space
St( d 1 , n ) × . . .
×
St( d l , n ) under a continuous function.
Define h : Z ⊆ R n 2 × l → R n by
[Equation]
Then note that h is continuous and permutation equivariant from matrices to vectors, so it can be ϵ -approximated by an invariant graph network (Keriven & Peyr ´ e, 2019), call it (cid:103) IGN .
If we define ˜ ρ = (cid:103) IGN and IGN d i ( V i V ⊤ i ) =
V i
V ⊤ i (this identity operation is linear and permutation equivariant, so it can be exactly expressed by an IGN), then we have ϵ -approximation of f by
[Equation]
E. P ROOF OF U NIVERSAL A PPROXIMATION FOR G ENERAL D ECOMPOSITIONS
Theorem 2 (Universal Approximation) .
Consider the same setup as Theorem 1, where X i are also compact.
Let Φ i be a family of G i -invariant functions that universally approximate G i -invariant continuous functions
X i
→ R a i , and let R be a set of continuous function that universally approximate continuous functions Z ⊆ R a → R s where a = (cid:80) i a i for every compact Z .
Then for any ε > 0 and any G -invariant continuous function f : X 1 × . . .
×
X k
→ R s there exists ϕ ∈ Φ and ρ ∈ R such that ∥ f − ρ ( ϕ 1 , . .
. , ϕ k )
∥ ∞ < ε .
Proof.
Consider a particular G -invariant continuous function f : X 1 × . . .
×X k
→ R s .
By Theorem 1 there exists G
i -invariant continuous functions ϕ
i : X i → R a i and a continuous function ρ ′ : Z ⊆ R a → R s (where a = (cid:80) i a i ) such that
[Equation]
Now fix an ε > 0 .
For any ρ ∈ R and any ϕ i ∈
Φ i ( i = 1 , . . .
k ) we may bound the difference from f as follows (suppressing the v
i ’s for brevity),
[Equation]
Now let K ′
= (cid:81) ki
=
im ϕ ′
i .
Since each ϕ
i is continuous and defined on a compact set
X i we know that im ϕ
i is compact, and so the product K is also compact.
Since K ′ is compact, it is contained in a closed ball B ( r ) of radius r > 0 centered at the origin.
Let K be the closed ball B ( r + 1) of radius r + 1 centered at the origin, so K contains K ′ and a ball of radius 1 around each point of K ′ .
We may extend ρ ′ continuously to K as needed, so assume ρ ′ :
K → R s .
By universality of R we may pick a particular ρ : K → R s , ρ ∈ R such that
[Equation]
Keeping this choice of ρ , it remains only to bound II.
As ρ is continuous on a compact domain, it is in fact uniformly continuous.
Thus, we can choose a δ ′ > 0 such that if ∥ y − z ∥ 2 ≤ δ ′ , then ∥ ρ ( y ) − ρ ( z ) ∥ ∞ < ϵ/ 2 , and then we define δ = min( δ ′ , 1) .
Since Φ i universally approximates ϕ ′
i we may pick ϕ i ∈
Φ i such that ∥ ϕ
i − ϕ ′ i
∥ ∞ < δ/ √ k , and thus ∥ ( ϕ 1 , . .
. , ϕ k ) − ( ϕ ′ 1 , . . .
ϕ ′ k ) ∥ ∞ ≤ δ .
With this choice of ϕ i , we know that (cid:81) ki
=
im ϕ
i ⊆ K (because each ϕ i ( x i ) is within distance 1 of ϕ
i ( x i ) ).
Thus, ρ ( ϕ 1 ( x 1 ) , . . . , ϕ k ( x k )) is well-defined, and we have
[Equation]
due to our choice of δ , which completes the proof.
F B ASIS I NVARIANCE FOR G RAPH R EPRESENTATION L EARNING F. S PECTRAL G RAPH C ONVOLUTION
In this section, we consider spectral graph convolutions, which for node features X ∈ R n × q take the form f ( V, Λ , X )
= (cid:80) ni
=
θ i v i v ⊤
i X for some parameters θ i .
We can optionally take θ i
= h ( λ i ) for some continuous function h : R → R of the eigenvalues.
This form captures most popular spectral graph convolutions in the literature (Bruna et al., 2014; Hamilton, 2020; Bronstein et al., 2017); often, such convolutions are parameterized by taking h to be some analytic function such as a simple affine function (Kipf & Welling, 2017), a linear combination in a polynomial basis (Defferrard et al., 2016; Chien et al., 2021), or a parameterization of rational functions (Levie et al., 2018; Bianchi et al., 2021).
First, it is well known and easy to see that spectral graph convolutions are permutation equivariant, as for a permutation matrix P we have
[Equation]
Also, it is easy to see that they are sign invariant, as ( − v i )( − v i )
⊤
= v i v ⊤ i .
However, if the θ i do not depend on the eigenvalues, then the spectral graph convolution is not necessarily basis invariant.
For instance, if v 1 and v 2 are in the same eigenspace, and we change basis by permuting v ′
= vand v ′
= v 1 , then if θ 1 ̸
= θ 2 the spectral graph convolution will generally change as well.
On the other hand, if θ i = h ( λ i ) for some function h : R → R , then the spectral graph convolution is basis invariant.
This is because if v i and v j belong to the same eigenspace, then λ i = λ
j so h
( λ i ) =
h ( λ j ) .
Thus, if v i 1 , . .
. , v i d are eigenvectors of the same eigenspace with eigenvalue λ , we have that (cid:80) dl
=1 h ( λ i l ) v i l v ⊤ i l = h ( λ ) (cid:80) dl
=1 v i l v ⊤ i l .
Now, note that (cid:80) dl
=1 v i l v ⊤ i l is the orthogonal projector onto the eigenspace (Trefethen & Bau III, 1997).
A change of basis does not change this orthogonal projector, so such spectral graph convolutions are basis invariant.
Another way to see this basis invariance is through basic algebra.
Let V 1 , . .
, V l be the eigenspaces of dimension d 1 , . . .
, d l , where V i ∈ R n × d i .
Let the corresponding eigenvalues be µ 1 , . . .
, µ l .
Then for any orthogonal matrices Q i ∈ O
( d i ) , we have
[Equation]
so the spectral graph convolution is invariant to substituting V j Q j for V j .
Now, we give the proof that shows SignNet and BasisNet can universally approximate spectral graph convolutions.
Proposition 1 (Learning Spectral Graph Convolutions) .
Suppose the node features X ∈ R n × q take values in compact sets.
Then SignNet can universally approximate any spectral graph convolution, and both BasisNet and Expressive-BasisNet can universally approximate any parametric spectral graph convolution.
Proof.
Note that eigenvectors and eigenvalues of normalized Laplacian matrices take values in compact sets, since the eigenvalues are in [0 , 2] and we take eigenvectors to have unit-norm.
Thus, the whole domain of the spectral graph convolution is compact.
Let ε > 0 .
First, consider a spectral graph convolution f ( V, Λ , X ) =
(cid:80) ni
=
θ i v i v ⊤
i X .
For SignNet, let ϕ ( v i , λ i , X ) approximate the function ˜ ϕ
( v i , λ i , X ) =
θ i v i v ⊤
i X to within ε/n error, which
DeepSets can do since this is a continuous permutation equivariant function from vectors to vectors (Segol & Lipman, 2019)
(note that we can pass λ
i as a vector in R n by instead passing λ i 1 , where 1 is the all ones vector).
Then ρ = (cid:80) ni
=1 is a linear permutation equivariant operation that can be exactly expressed by DeepSets, so the total error is within ε .
The same argument applies when θ i
= h ( λ i ) for some continuous function h .
For the basis invariant case, consider a parametric spectral graph convolution f ( V, Λ , X )
=
(cid:80) n i =1 h ( λ i ) v i v ⊤ i X .
Note that if the eigenspace bases are V 1 , . .
, V l with eigenvalues µ 1 , . . .
, µ l , we can write the f ( V, Λ , X ) =
(cid:80) li =1 h ( µ j ) V j V ⊤ j X .
Again, we will let ρ = (cid:80) li
=1 be a sum function, which can be expressed exactly by DeepSets.
Thus, it suffices to show that h ( µ j ) V j V ⊤
j X can be ϵ/n approximated by a 2-IGN (i.e. an IGN that only uses vectors and matrices).
Note that since h is continuous, we can use an elementwise MLP (which IGNs can learn) to approximate f 1 ( µ 11 ⊤ , V V ⊤ , X ) =
( h ( µ ) 11 ⊤ , V V ⊤ , X ) to arbitrary precision (note that we represent the eigenvalue µ as a constant matrix µ 11 ⊤ ).
Also, since a 2-IGN can learn matrix vector multiplication (Cai & Wang (2022) Lemma 10), we can approximate f 2 ( h ( µ ) 11 ⊤ , V V ⊤ , X ) =
( h ( µ ) 11 ⊤ , V V ⊤ X ) , as V i V ⊤ i ∈ R n 2 is a matrix and X ∈ R n × q is a vector with respect to permutation symmetries.
Finally, we use an elementwise MLP to approximate the scalar-vector multiplication f 3 ( h ( µ ) 11 ⊤ , V V ⊤ , X )
=
h ( µ ) V V ⊤ X .
Since f 3 ◦ f 2 ◦ f 1 ( µ 11 ⊤ , V V ⊤ , X ) = h ( µ ) V V ⊤ X , and since 2-IGNs universally approximate each f i , applying Lemma 6 shows that a 2-IGN can approximate h ( µ ) V V ⊤ X to ϵ/n accuracy, so we are done.
Since Expressive-BasisNet is stronger than
BasisNet, it can also universally approximate these functions.
From the proof, we can see that SignNet and BasisNet need only learn simple functions for the ρ and ϕ when h is simple, or when the filter is non-parametric and we need only learn θ i .
Xu et al.
(2020) propose the principle of algorithmic alignment, and show that if separate modules of a neural network each need only learn simple functions (that is, functions that are well-approximated by loworder polynomials with small coefficients), then the network may be more sample efficient.
If we do not require permutation equivariance, and parameterize SignNet and BasisNet with simple MLPs, then algorithmic alignment may suggest that our models are sample efficient.
Indeed, ρ = (cid:80) is a simple linear function with coefficients 1 , and ϕ ( V, λ, X ) =
h ( λ ) V V ⊤ X is quadratic in V and linear in X , so it is simple if h is simple.
F. E XISTING P OSITIONAL E NCODINGS
Here, we show that our SignNets and BasisNets universally approximate various types of existing graph positional encodings.
The key is to show that these positional encodings are related to spectral graph convolution matrices and diagonals, and to show that our networks can approximate these matrices and diagonals.
Proposition 6.
SignNets and BasisNets universally approximate the diagonal of any spectral graph convolution matrix f
( V, Λ) =
diag (cid:0)(cid:80) ni
=1 h ( λ i ) v i v ⊤ i (cid:1) .
BasisNets can additionally universally approximate any spectral graph convolution matrix f
( V, Λ) = (cid:80) n i =1 h ( λ i ) v i v ⊤ i .
Proof.
Note that v i
and λ i come from compact sets, as λ i ∈ [0 , 2] for the normalize Laplacian
and v i is of unit norm.
Also, as diag is linear, the spectral graph convolution diagonal can be written (cid:80) ni
=1 h ( λ i )diag( v i v ⊤ i ) .
Let ϵ > 0 .
For SignNet, let ρ = (cid:80) ni =1 , which can be exactly expressed as it is a permutation equivariant linear operation from vectors to vectors.
Then ϕ ( v i , λ i ) can approximate the function λ i diag( v i v ⊤ i ) to arbitrary precision, as it is a permutation equivariant function from vectors to vectors (Segol & Lipman, 2019).
Thus, letting ϕ approximate the function to ϵ/n accuracy, SignNet can approximate f to ϵ accuracy.
Let l be the number of eigenspaces V 1 , . .
, V l , so f ( V, Λ) =
(cid:80) li =1 h ( µ i ) V
i V ⊤ i .
For BasisNet, we need only show that it can approximate the spectral graph convolution matrix to ϵ/l accuracy, as a 2-IGN can exactly express the diag function in ρ , since it is a linear permutation equivariant function from matrices to vectors.
A 2-IGN can universally approximate the function f 1 ( µ i , V i V ⊤ i )
=
( h ( µ i ) , V i V ⊤ i ) , as it can express any elementwise MLP.
Also, a 2-IGN can universally approximate the scalar-matrix multiplication f 2 ( h ( µ i ) , V i V ⊤ i )
=
h ( µ i ) V
i V ⊤ i by another elementwise MLP.
Since h ( µ i ) V i V ⊤ i =
f 2 ◦ f 1 ( µ i , V i V ⊤ i )
, Lemma 6 shows that a single 2-IGN can approximate this composition to ϵ/l accuracy, so we are done.
Proposition 7. SignNets and BasisNets universally approximate heat kernel positional encodings (Feldman et al., 2022) and random walk node positional encodings (RWPE) (Dwivedi et al., 2022).
BasisNets universally approximate diffusion and p -step random walk relative positional encodings (Mialon et al., 2021), as well as generalized PageRank and landing probability distance encodings (Li et al., 2020).
Proof.
We will show that we can apply the above Proposition 6, by showing that all of these positional encodings are spectral graph convolutions.
The heat kernel embeddings are of the form diag (cid:0)(cid:80) ni =1 exp( − tλ i ) v i v ⊤ i (cid:1) for some choices of the parameter t , so they can be approximated by
SignNets or BasisNets.
Also, the diffusion kernel (Mialon et al., 2021) is just the matrix of this heat kernel, and the p -step random walk kernel is (cid:80) n i =1 (1 − γλ i ) p
v i v
⊤ i for some parameter γ , so
BasisNets can universally approximate both of these.
For the other positional encodings, we let v i be the eigenvectors of the random walk Laplacian
I − D − 1 A instead of the normalized Laplacian
I − D − 1 / 2 AD − 1 / 2 .
The eigenvalues of these two
Laplacians are the same, and if ˜ v i is an eigenvector of the normalized Laplacian then D − 1 / 2 ˜ v
i is an eigenvector of the random walk Laplacian with the same eigenvalue (Von Luxburg, 2007).
Then with v i as the eigenvectors of the random walk Laplacian, the random walk positional encodings (RWPE) in Dwivedi et al.
(2022) take the form
[Equation]
for any choices of integer k .
The distance encodings proposed in Li et al.
(2020) take the form
[Equation]
for some function f 3 .
We restrict to continuous f 3 here; shortest path distances can be obtained by a discontinuous f 3 that we discuss below.
Their generalized PageRank based distance encodings can be obtained by
[Equation]
for some γ k ∈ R , so this is a spectral graph convolution.
They also define so-called landing probability based positional encodings, which take the form
[Equation]
for some choices of integer k .
Thus, BasisNets can approximate these distance encoding matrices.
Another powerful class of positional encodings is based on shortest path distances between nodes in the graph (Ying et al., 2021; Li et al., 2020).
Shortest path distances can be expressed in a form similar to the spectral graph convolution, but require a highly discontinuous function.
If we define φ ( x 1 , . . .
, x n )
=
min
i : x i ̸ =
i to be the lowest index such that x i is nonzero, then we can write the shortest path distance matrix as φ ( D − 1 A, ( D − 1 A ) 2 , . . .
, ( D − 1 A ) n ) , where φ is applied elementwise to return an n × n matrix.
As ( D − 1 A ) k
= (cid:80) ni
=1 (1 − λ i )
k
v i v ⊤ i , BasisNets can learn the inside arguments, but cannot learn the discontinuous function φ .
F. S PECTRAL I NVARIANTS
Here, we consider the graph angles α ij =
∥ V i V ⊤ i e j ∥ 2 , for i = 1 , . . .
, l where l is the number of eigenspaces, and j = 1 , . . .
, n .
It is clear that graph angles are permutation invariant and basis invariant.
These graph angles have been extensively studied, so we simply cite a number of interesting properties of them.
That graph angles determine the number of length 3, 4 and 5 cycles, the connectivity of a graph, and the number of length k closed walks is all shown in Chapter 4 of Cvetkovi ´ c et al.
Other properties may be of use for graph representation learning as well.
For instance, the eigenvalues of node-deleted subgraphs of a graph G are determined by the eigenvalues and graph angles of G ; this may be useful in extending recent graph neural networks that are motivated by node deletion and the reconstruction conjecture (Cotta et al., 2021; Bevilacqua et al., 2022; Papp et al., 2021; Tahmasebi et al., 2020).
Now, we prove that BasisNet can universally approximate the graph angles.
The graph properties we consider in the proposition are all integer valued (e.g. the number of cycles of length 3 in a graph is an integer).
Thus, any two graphs that differ in these properties will differ by at least 1, so as long as we have universal approximation to ε < 1 / 2 , we can distinguish any two graphs that differ in these properties.
Recall the statement of Proposition 2.
Proposition 2.
BasisNet can universally approximate the graph angles α ij .
The eigenvalues and graph angles (and thus BasisNets) can determine the number of length 3, 4, and 5 cycles, whether a graph is connected, and the number of length k closed walks from any vertex to itself.
Proof.
Note that the graph angles satisfy
[Equation]
where V i is a basis for the i th adjacency matrix eigenspace, and
e ⊤ j V i V ⊤ i
e j is the ( j, j ) -entry of
V i V ⊤ i .
These graph angles are just the elementwise square roots of the diagonals of the matrices
V i V ⊤ i .
As f 1 ( V i V ⊤ i ) =
diag( V i V ⊤ i ) is a permutation equivariant linear function from matrices to vectors, 2-IGN on V
i V ⊤ i can exactly compute this with 0 error.
Then a 2-IGN can learn an elementwise MLP to approximate the elementwise square root f 2 (diag( V i V ⊤ i ))
= (cid:112) diag( V i V ⊤ i ) to arbitrary precision.
Finally, there may be remaining operations f 3 that are permutation invariant or permutation equivariant from vectors to vectors; for instance, the α ij are typically gathered into a matrix of size l × n where the columns are lexicographically sorted ( l is the number of eigenspaces) (Cvetkovi´c et al., 1997), or we may have a permutation invariant readout to compute a subgraph count.
A DeepSets can approximate f 3 without any higher order tensors besides vectors (Zaheer et al., 2017; Segol & Lipman, 2019).
As 2-IGNs can approximate each f i individually, a single 2-IGN can approximate f 3 ◦ f 2 ◦ fby Lemma 6.
Also, since the graph properties considered in the proposition are integer-valued,
BasisNet can distinguish any two graphs that differ in one of these properties.
G U SEFUL L EMMAS
In this section, we collect useful lemmas for our proofs.
These lemmas generally only require basic tools to prove.
Our first lemma is a crucial property of quotient spaces.
Lemma 1 (Passing to the quotient) .
Let X and Y be topological spaces, and let X /G be a quotient space, with corresponding quotient map π .
Then for every continuous G -invariant function f : X →
Y , there is a unique continuous ˜ f : X /G
→ Y such that f = ˜ f ◦ π .
Proof.
For z ∈ X /G , by surjectivity of π we can choose an x z ∈ X such that π ( x z ) =
z .
Define ˜ f
: X /G → Y by ˜ f ( z ) =
f ( x z ) .
This is well-defined, since if π ( x z ) =
π ( x ) for any other x ∈ X , then gx z = x for some g ∈ G , so
[Equation]
[Equation] where the second equality uses the G -invariance of f .
Note that ˜ f is continuous by the universal property of quotient spaces.
Also, ˜ f is the unique function such that f = ˜ f ◦ π ; if there were another function h : X /G
→ Y with h ( z ) ̸ =
˜ f
( z ) , then h ( z ) ̸
= f ( x z ) , so h ( π ( x z ))
=
h ( z ) ̸
= f ( x z ) .
Next, we give the First Fundamental Theorem of O ( d ) , a classical result that has been recently used for machine learning by Villar et al.
This result shows that an orthogonally invariant f ( V ) can be expressed as a function h ( V V ⊤ ) .
We give a proof that if f is continuous, then h is also continuous.
Lemma 2 (First Fundamental Theorem of O ( d ) ) .
A continuous function f : R n × d → R s is orthgonally invariant, i.e. f ( V Q ) =
f ( V ) for all Q ∈ O ( d ) , if and only if f ( V )
=
h ( V V ⊤ ) for some continuous h .
Proof.
If f ( V ) =
h ( V V ⊤ ) , then we have f ( V Q ) =
h ( V QQ ⊤ V ⊤ ) =
h ( V V ⊤ ) so f is orthogonally invariant.
For the other direction, invariant theory shows that the O ( d ) invariant polynomials are generated by the inner products v ⊤ i v j , where v i ∈ R
d are the rows of V (Kraft & Procesi, 1996).
Let p : R n × d →
R n × n be the map p ( V ) =
V V ⊤ .
Then Gonz´alez & de Salas (2003)
Lemma 11.13 shows that the quotient space R n × d /O ( d ) is homeomorphic to a closed subset p ( R n × d )
=
Z ⊆ R n × n , let ˜ p refer to this homeomorphism, and note that ˜ p ◦ π = p by passing to the quotient (Lemma 1).
Then any continuous O ( d ) invariant f passes to a unique continuous ˜ f
: R n × d /O ( d ) → R s (Lemma 1), so f = ˜ f ◦ π where π is the quotient map.
Define h : Z → R s by h = ˜
f ◦ ˜ p − 1 , and note that h is a composition of continuous functions and hence continuous.
Finally, we have that h ( V V ⊤ )
=
h (˜ p ◦ π ( V )) =
˜ f ◦ π ( V ) =
f ( V ) , so we are done.
The next lemma allows us to decompose a quotient of a product space into a product of smaller quotient spaces.
Lemma 3.
Let X 1 , . . .
, X k be topological spaces and G 1 , . . .
, G k be topological groups such that each G i acts continuously on X i .
Denote the quotient maps by π i : X i → X i /G i .
Then the quotient of the product is the product of the quotient, i.e.
[Equation]
and π 1 × . . .
×
π k : X 1 × . . .
X k
→ ( X 1 /G 1 ) × . . .
× ( X k /G k ) is quotient map.
Proof.
First, we show that π 1 × . . .
×
π k is a quotient map.
This is because 1.
the quotient map of any continuous group action is an open map, so each π i is an open map, 2.
the product of open maps is an open map, so π 1 × . . .
×
π k is an open map and 3.
a continuous surjective open map is a quotient map, so π 1 × . . .
× π k , which is continuous and surjective, is a quotient map.
Now, we need only apply the theorem of uniqueness of quotient spaces to show 40 (see e.g. Lee (2013), Theorem A.31).
Letting q : X 1 × . . .
×
X k → ( X 1 × . . .
×
X k ) /
( G 1 × . . .
× G k ) denote the quotient map for this space, it is easily seen that q ( x 1 , . . .
, x k ) =
q ( y 1 . . .
, y k )
if and only if π 1 × . . .
× π k ( x 1 , . . .
, x k ) =
π 1 × . . .
× π k
( y 1 , . . .
, y k ) , since either of these is true if and only if there exist g i ∈
G i such that x i =
g
i
y
i for each i .
Thus, we have an isomorphism of these quotient spaces.
The following lemma shows that quotients of compact spaces are also compact, which is useful for universal approximation on quotient spaces.
Lemma 4 (Compactness of quotients of compact spaces) .
Let X be a compact space.
Then the quotient space X /G is compact.
Proof.
Denoting the quotient map by π : X → X /G and letting { U α } α be an open cover of X /G , we have that { π − 1 ( U α ) } α is an open cover of X .
By compactness of X , we can choose a finite subcover { π − 1 ( U α i ) } i =1 ,...,n .
Then { π ( π − 1 ( U α i ))
} i =1 ,...,n = { U α i } i =1 ,...,n by surjectivity, and { U α i } i =1 ,...,n is thus an open cover of X /G .
The Whitney embedding theorem gives a nice condition that we apply to show that the quotient spaces X /G that we deal with embed into Euclidean space.
It says that when X /G is a smooth manifold, then it can be embedded into a Euclidean space of double the dimension of the manifold.
The proof is outside the scope of this paper.
Lemma 5 (Whitney Embedding Theorem (Whitney, 1944)) .
Every smooth manifold M of dimension n > 0 can be smoothly embedded in R 2 n .
Finally, we give a lemma that helps prove universal approximation results.
It says that if functions f that we want to approximate can be written as compositions f
= f L ◦ . . .
◦ f 1 , then it suffices to universally approximate each f i and compose the results to universally approximate the f .
This is especially useful for proving universality of neural networks, as we may use some layers to approximate each f i , then compose these layers to approximate the target function f .
Lemma 6 (Layer-wise universality implies universality) .
Let Z ⊆
R
d 0 be a compact domain, let
F 1 , . .
. , F L be families of continuous functions where F i consists of functions from R d i − 1 → R d i for some d 1 , . . .
, d L .
Let F be the family of functions { f L ◦ . . .
f 1 : Z → R d L , f i ∈ F i } that are compositions of functions f i ∈ F i .
For each i , let Φ i be a family of continuous functions that universally approximates F i .
Then the family of compositions Φ = { ϕ L ◦ . . .
◦
ϕ 1 : ϕ i ∈
Φ i } universally approximates F .
Proof.
Let f = f L ◦ . . .
◦ f 1 ∈ F .
Let ˜ Z
=
Z , and then for i ≥ 2 let ˜ Z i
=
f i −
( ˜ Z i − 1 ) .
Then each ˜ Z i is compact by continuity of the f i .
For 1 ≤ i < L , let Z i =
˜
Z i , and for i =
L let Z L be a compact set containing ˜ Z L such that every ball of radius one centered at a point in ˜ Z L is still contained in Z L .
Let ϵ > 0 .
We will show that there is a ϕ ∈ Φ such that ∥ f − ϕ ∥ ∞ <
ϵ by induction on L .
This holds trivially for L = 1 , as then Φ =
Φ 1 .
Now, let L ≥ 2 , and suppose it holds for L − 1 .
By universality of Φ L , we can choose a ϕ L : Z L → R d L ∈ Φ
L such that ∥ ϕ
L − f L ∥ ∞ < ϵ/ 2 .
As ϕ L is continuous on a compact domain, it is also uniformly continuous, so we can choose a ˜ δ > 0 such that ∥ y − z ∥ 2 < ˜ δ = ⇒ ∥
ϕ L ( y ) − ϕ L ( z ) ∥ 2 < ϵ/ 2 .
Let δ = min(˜ δ, 1) .
By induction, we can choose ϕ L − 1 ◦ . . .
◦
ϕ 1 , ϕ i ∈
Φ i such that
[Equation]
Note that ϕ L − 1 ◦ . . .
◦ ϕ 1 ( Z ) ⊆
Z L , because for each x ∈ Z , ϕ L − 1 ◦ . . .
◦
ϕ 1 ( x ) is within δ ≤
Euclidean distance to f L − 1 ◦ . . .
◦ f 1 ( x ) ∈ ˜
Z L , so it is contained in Z L by construction.
Thus, we may define ϕ = ϕ
L ◦ . . .
◦
ϕ 1 : Z → R d L , and compute that
[Equation] [Equation]
43) since ∥ ϕ L − f L ∥ ∞ < ϵ/ 2 .
To bound this other term, let x ∈ Z , and for y =
ϕ
L − 1 ◦ . . .
◦
ϕ 1 ( x ) and z = f
L − 1 ◦ . . .
◦ f 1 ( x ) , we know that ∥ y − z ∥ 2 < δ , so ∥ ϕ L ( y )
− ϕ L ( z ) ∥ 2 < ϵ/ 2 by uniform continuity.
As this holds for all x , we have ∥ ϕ − ϕ
L ◦
f L − 1 ◦ . . .
◦
f 1 ∥ ∞ ≤ ϵ/ 2 , so ∥ ϕ − f ∥ ∞ <
ϵ
and we are done.
H C OUNTING S UBSTRUCTURES AND R EGRESSING G RAPH P ROPERTIES
PEs, SignNet improves performance, while sign flip data augmentation is less consistent.
Mean and standard deviations are reported on 3 runs.
The same 4-layer GIN is used across all settings for fair comparison.
[Table]
Substructure counts (e.g. of cycles) and global graph properties (e.g. connectedness, diameter, radius) are important graph features that are known to be informative for problems in bio- and chemo-informatics (Chen et al., 2020; Corso et al., 2020).
Following the setting in Zhao et al.
(2022), we show that SignNet with Laplacian positional encodings boosts the ability of simple GNNs to count substructures and regress graph properties.
We take a 4-layer GIN as the base model for all settings, and for SignNet we use GIN as ϕ and a Transformer as ρ to handle variable numbers of eigenvectors (see Appendix.
I.3 for details).
As shown in Table 5, Laplacian PEs with flipbased data augmentation improves performance in counting substructures but not in regressing graph properties, while Laplacian PEs processed by SignNet significantly boost performance on all tasks.
H. I MPLICIT N EURAL R EPRESENTATIONS ON M ANIFOLDS
Discrete approximations to the Laplace-Beltrami operator on manifolds have proven useful for processing data on surfaces, such as triangle meshes (L´evy, 2006).
Recently, Koestler et al.
(2022) propose intrinsic neural fields, which use eigenfunctions of the Laplace-Beltrami operator as positional encodings for learning implicit neural representations on manifolds.
For generalized eigenfunctions v 1 , . . .
, v k , at a point p on the surface, they parameterize functions f ( p ) =
MLP( v 1 ( p ) , . .
. , v k ( p )) .
As these eigenfunctions have sign ambiguity, we use our SignNet to parameterize f ( p ) =
MLP( ρ ([ ϕ ( v i ( p ))
+ ϕ ( − v i ( p ))]
i =1 ,...,k )) , where ρ and ϕ are also MLPs.
Table 6 shows our results for texture reconstruction experiments on a cat model.
The total number of parameters in our SignNet-based model is kept below that of the original model.
We see that the SignNet architecture improves over the original Intrinsic NF model, as well as over models that take in the absolute values or sign flipped eigenfunctions.
While we have not tested this, we believe that SignNet would allow even better improvements when learning over eigenfunctions of different models, as it could improve transfer and generalization.
See Appendix I.4 for more details.
[Table]
I F URTHER E XPERIMENTAL D ETAILS I. S PECTRAL G RAPH C ONVOLUTION D ETAILS
In Section 4, we conduct node regression experiments for learning spectral graph convolutions.
The experimental setup is mostly taken from He et al.
However, we take the dataset of 50 images in He et al.
(2021) (originally from the Image Processing Toolbox of M ATLAB ), and resize them from 100 × 100 to 32 × 32.
Thus, each image is viewed as a 1024 -node graph.
The node features
X ∈ R n are the grayscale pixel intensities of each node.
Then we apply the same spectral graph convolutions on them as in He et al.
(2021), and train neural networks to learn these as regression targets.
As in prior work, we report sum of squared errors on the training set to test expressivity.
Just as in He et al.
(2021), we only train and evaluate on nodes that are not connected to the boundary of the grid (that is, we only evaluate on the 28 × 28 middle section).
For all experiments we limit each model to 50,000 parameters.
For each of the GNN baselines (GPR-GNN, ARMA, ChebNet,
BernNet), we select the best performing out of 4 hyperparameter settings: either 2 or 4 convolution layers, and a hidden dimension of size 32 or D , where D is just large enough to stay with 50, parameters (for instance, D = 128 for BernNet).
We use DeepSets or standard Transformers as our prediction network.
This takes in the output of
SignNet or BasisNet and concatenates it with the node features, then outputs a scalar prediction.
We use a 3 layer output network for DeepSets SignNet, and 2 layer output networks for all other configurations.
All networks use ReLU activations.
For SignNet, we use DeepSets for both ϕ and ρ .
Our ϕ takes in eigenvectors only, then our ρ takes the outputs of ϕ and the eigenvalues.
We use three layers for ϕ and ρ .
For BasisNet, we use the same DeepSets for ρ as in SignNet, and 2-IGNs for the ϕ d i .
There are three distinct multiplicities for the grid graph (1, 2, and 32), so we only need 3 separate IGNs.
Each
IGN consists of an R n 2 × 1 → R n × d ′ layer and two R n × d ′′
→ R n × d ′′′ layers, where the d ′ are hidden dimensions.
There are no matrix to matrix operations used, as the memory requirements are intensive for these ≥ 1000 node graphs.
The ϕ d i only take in V
i V ⊤ i from the eigenspaces, and the ρ takes the output of the ϕ d i as well as the eigenvalues.
I. G RAPH R EGRESSION D ETAILS
We parameterize the SignNet by taking ϕ to be 8 layer GIN with ReLU activation (Xu et al., 2019) and ρ to be an 8 layer MLP with batch normalization and ReLU activation.
We consider four different base models: GatedGCN (Bresson & Laurent, 2017), a Transformer with sparse attention only over neighbours as introduced by Kreuzer et al.
(2021), PNA (Corso et al., 2020), and GIN
[Table]
The input eigenvector v i ∈ R n , where n is the number of nodes in the graph, is treated as a single scalar feature for each node.
For a fixed number k of eigenvectors, the aggregator ρ is applied separately to the concatenatation of the k different embeddings for each node in a graph, resulting in one single embedding per node.
When using all eigenvectors, since the number of eigenvectors varies across graphs we sum the outputs of ϕ ( v ) + ϕ ( − v ) for each node, then process this sum with ρ .
This embedding is concatenated to the node features for that node, and the result passed as input to the base (predictor) model.
Our training pipeline largely follows that of Dwivedi et al.
, and we use the GatedGCN and PNA base models from the accompanying implementation (see https://github.com/ vijaydwivedi75/gnn-lspe ).
The Sparse Transformer base model architecture we use, which like GAT computes attention only across neighbouring nodes, is introduced by Kreuzer et al.
Finally, the GINE implementation is based on the PyTorch Geometric implementation (Fey &
Lenssen, 2019).
All models in Table 1 use edge features for learning and inference.
To show that SignNet is also useful when no edge features are available, we ran ZINC experiments without edge features as well.
The results are displayed in Table 7.
I. S UBSTRUCTURES AND G RAPH P ROPERTIES R EGRESSION D ETAILS
We use the random graph dataset from Chen et al.
(2020) for counting substructures and the synthetic dataset from Corso et al.
(2020) for regressing graph properties.
To keep fair comparison we fix the base model as a 4-layer GIN model with hidden size 128.
As graphs can have different number of nodes, the number of Laplacian eigenvectors is the same as the number of nodes and hence is different across graphs.
One can truncate it to keep the top k most important eigenvectors but this loses information.
We may instead use all eigenvectors by designing the model to handle positional encodings of varying sizes.
SignNet can accept varying numbers of Laplacian eigenvectors as input by choosing ρ and ϕ to be neural networks allowing variable-size input.
Specifically, we choose ϕ as 4-layer GIN (independently applied to every eigenvector) and ρ as 1-layer Transformer (independently applied to every node).
Combined with proper batching and masking, we have a SignNet that takes Laplacian eigenvectors V ∈ R n × n and outputs fixed size sign-invariant encoding node features f ( V, Λ , X ) ∈ R n × d , where n varies between graphs but d is fixed.
We use this SignNet in our experiments and compare with other methods of handling PEs.
I. T EXTURE R ECONSTRUCTION D ETAILS
We closely follow the experimental setting of Koestler et al.
for the texture reconstruction experiments.
In this work we only test on the cat model, and we use the cotangent Laplacian (L´evy, 2006) of a triangle mesh with the lowest k eigenvectors (for k ∈ { 64 , 1023 } ) besides the eigenvector of eigenvalue 0.
We implemented SignNet in the authors’ original code, which was privately shared with us.
Both ρ and ϕ are taken to be MLPs.
Hyperparameter settings and number of parameters are given in Table 8.
We chose hyperparameters so that the number of total parameters in the SignNet model was no larger than that of the original model.
[Table]