W EISFEILER AND L EHMAN G O T OPOLOGICAL : M ESSAGE P ASSING S IMPLICIAL N ETWORKS
A BSTRACT
The pairwise interaction paradigm of graph machine learning has predominantly governed the modelling of relational systems.
However, graphs alone cannot capture the multi-level interactions present in many complex systems and the expressive power of such schemes was proven to be limited.
To overcome these limitations, we propose Message Passing Simplicial Networks (MPSNs), a class of models that perform message passing on simplicial complexes (SCs) – topological objects generalising graphs to higher dimensions.
To theoretically analyse the expressivity of our model we introduce a Simplicial Weisfeiler-Lehman (SWL) colouring procedure for distinguishing non-isomorphic SCs.
We relate the power of
SWL to the problem of distinguishing non-isomorphic graphs and show that SWL and MPSNs are strictly more powerful than the WL test and not less powerful than the 3-WL test.
We deepen the analysis by comparing our model with traditional graph neural networks with ReLU activations in terms of the number of linear regions of the functions they can represent.
We empirically support our theoretical claims by showing that MPSNs can distinguish challenging strongly regular graphs for which GNNs fail and, when equipped with orientation equivariant layers, they can improve classiﬁcation accuracy in oriented SCs compared to a GNN baseline.
Our model also attains competitive results on real-world graph classiﬁcation datasets, with best performance on those tasks with a more prominent number of higher-order interactions.
Additionally, we implement a library for neural message passing on simplicial complexes that we envision to release in due course.
I NTRODUCTION
vvv 6 vvvvvvvv
Graph neural networks (GNNs) have been well developed for learning features of graphs, which are one of the most common abstractions for complex systems (Bronstein et al., 2017).
However, GNNs are limited in their capability of capturing higher-order interactions such as triangles or cliques (Chen et al., 2020) that are usually prominent in many natural graphs such as organic molecules (Bouritsas et al., 2020)
(e.g. aromatic rings) or social networks (Milo et al., 2002).
This paper tries to enhance GNN expressivity by considering local higher-order interactions.
Among many modelling frameworks that have been proposed to describe complex systems with higher-order relations (Battiston et al., 2020), we speciﬁcally focus on simplicial complexes, a convenient middle ground between graphs (which are a particular case of a simplicial complex) and more general hypergraphs.
Importantly, they offer strong mathematical connections to algebraic and differential topology and geometry.
The simplicial
Hodge Laplacian (Barbarossa & Sardellitti, 2020; Schaub et al., 2020), a discrete counterpart of the
Laplacian operator in Hodge–de Rham theory (Rosenberg, 1997), provides a connection with the theory of spectral analysis and signal processing on these higher-dimensional domains.
We start by deriving from ﬁrst principles a Simplicial Weisfeiler-Lehman (SWL) test for distinguishing non-isomorphic simplicial complexes.
Motivated by this theoretical construction, we propose
Message Passing Simplicial Networks (MPSNs), a message passing neural architecture for simplicial complexes that extends previous approaches such as GNNs and spectral simplicial convolutions (Bunch et al., 2020;
Ebli et al., 2020).
We then show that the proposed MPSN is as powerful as SWL.
Strictly better than the conventional WL test (Weisfeiler & Leman, 1968), MPSN can be used to distinguish non-isomorphic graphs.
We also show that the SWL test and MPSN are not less powerful than the 3-WL test (Cai et al., 1992; Morris et al., 2019).
Moreover, we explore the expressive power of GNNs and MPSNs in terms of the number of linear regions of the functions they can represent (Pascanu et al., 2013; Mont´ufar et al., 2014).
We obtain bounds for the maximal number of linear regions of MPSNs and show a higher functional complexity than GNNs and simplicial convolutional neural networks (SCNNs) (Ebli et al., 2020), for which we provide optimal bounds that might be of independent interest.
Proofs and the required background are presented in the Appendix.
We will also be releasing a library based on PyTorch Geometric (Fey & Lenssen, 2019) for neural message passing on simplicial complexes.
The library can handle SC datasets, simplicial message passing networks, oriented SCs, (higher-order) batching, clique complexes and many other features.
M ESSAGE P ASSING S IMPLICIAL N ETWORKS
Simplicial WL.
We develop a simplicial version of the WL test with the ultimate goal of deriving a message passing procedure that can retain the expressive power of the test.
We call this simplicial colouring algorithm Simplicial WL (SWL).
Given a complex K , all the simplices σ ∈ K are initialised with the same colour.
Given the colour c tσ of simplex σ at iteration t , we compute the colour of simplex σ at the next iteration c t +1 σ , by perfectly hashing the multi-sets of colours belonging to the adjacent simplices of σ .
The algorithm stops after a ﬁnite number of steps or when the colours are no longer updated.
Two simplicial complexes are considered non-isomorphic if the colour histograms at any level of the complex are different.
A crucial choice has to be made about what simplices are considered to be adjacent.
Unlike graphs, simplicial complexes contain many types of adjacencies: lower adjacencies (i.e. sharing of an immediate face), upper adjacencies (i.e. sharing of an immediate coface), boundary relationships (e.g. a triangle that is the face of a tetrahedron) and the opposite co-boundary relationships.
To circumvent making arbitrary decisions, we start with an SWL test that includes all these possible choices.
We use the following notation: c t F ( σ ) = {{ c tω | ω ∈ F ( σ ) }} for the colours of the faces of σ , c t C ( σ ) = {{ c tω | ω ∈ C ( σ ) }} for the colour of the cofaces, c t ↓ ( σ ) =
{{ ( c tω , c tσ ∩ ω ) | ω ∈ N ↓ ( σ ) }} for the colour of its down adjacent neighbours and the face σ ∩ ω they share, and ﬁnally, c t ↑ ( σ )
= {{ ( c tω , c tσ ∪ ω ) | ω ∈ N ↑ ( σ ) }} for the colours of the upper adjacent neighbours and the coface σ ∪ ω they share.
We then obtain the following update rule, which contains the complete set of adjacencies:
[Equation]
Starting from this complete set, we will now show that certain adjacencies can be removed without sacriﬁcing the expressive power of the test (in terms of simplicial complexes that can be distinguished).
Theorem 1.
SWL with c t +1 v = HASH (cid:0) c tv , c t F ( v ) ,
c t ↑ ( v ) (cid:1) is as powerful as SWL with the generalised update rule HASH (cid:0) c tv , c t F ( v ) , c t C ( v ) , c t ↓ ( v ) , c t ↑ ( v ) (cid:1) .
We note that other possible combinations of adjacencies might also fully preserve the expressive power of the general SWL test.
The SWL procedure can be used not only to distinguish simplicial complexes, but also graphs.
To test the isomorphism of two graphs, SWL can test the isomorphism of the corresponding clique complexes of the graphs (i.e. every ( k + 1)
-clique in the graph becomes a k -simplex in the complex).
By taking this pre-processing step, we can relate SWL to WL:
Theorem 2.
SWL is strictly more powerful than WL.
We present in Figure 5 (Appendix) a pair of graphs that cannot be distinguished by the WL test, but whose clique complexes can be distinguished by SWL.
Theorem 3.
SWL is not less powerful than 3-WL.
MPSN.
We propose the following message passing operations based on the four types of messages discussed in the previous section: faces, cofaces, lower adjacencies and upper adjacencies.
[Equation]
If the underlying complex also has a particular orientation, this extra information can be easily included in the message functions deﬁned above.
Then, the update operation takes into account these four types of incoming messages and the previous colour of the simplex h t +1 v [Equation] U (cid:16) [Equation]
, m
[Equation] ( v ) , m t [Equation] ( v ) , m
[Equation]
+1 ↓ ( v ) [Equation] m t +1 ↑ ( v ) (cid:17) . To obtain a global embedding for a p -simplicial complex K from an MPSN with L layers, the readout function takes as input the multi-sets of colours corresponding to all the dimensions of the complex h G =
READOUT ( {{ h Lv }} v ∈K 0 , . .
. , {{ h Lv }} v ∈K p ) .
Lemma 4.
MPSNs are at most as powerful as SWL in distinguishing non-isomorphic simplicial complexes.
Theorem 5.
MPSNs with a sufﬁcient number of layers and injective message, aggregate and update functions are as powerful as SWL in distinguishing non-isomorphic simplicial complexes.
This theorem, combined with Theorem 2, provides an important corollary that MPSNs are not only beneﬁcial for statistical tasks on higher-dimensional simplicial complexes, but they can also improve over standard GNNs on graph machine learning tasks.
Corollary 6.
There exists an MPSN that is more powerful than WL at distinguishing non-isomorphic graphs when using a clique-complex lifting.
Theorem 7.
MPSNs generalise certain spectral convolution operators (Ebli et al., 2020; Bunch et al., 2020) deﬁned over simplicial complexes.
Next we show how MPSNs handle the symmetries present in simplicial complexes.
We defer the treatment of orientation equivariance and invariance in oriented SCs for Appendix E.
Theorem 8 (Informal) .
An MPSN layer is (simplex) permutation equivariant, while an MPSN network ending with a readout layer is (simplex) permutation invariant.
GNN
SCNN
MPSN - -0.
1 Regions over 2D slice ofGNN input space, S=3, d=1, m=3, p= - -0.
1 Regions over 2D slice ofSCNN input space, S=3, d=1, m=3, p= - -0.
1 Regions over 2D slice ofMPSN input space, S=3, d=1, m=3, p=
Linear regions analysis.
While the WL test has been used for studying the expressive power of
GNNs, other tools have been used to study the expressive power of conventional neural networks, like fully connected and convolutional.
One such tool is based on the number of linear regions of networks using piece-wise linear activations (Pascanu et al., 2013; Mont´ufar et al., 2014).
We show how this tool can also be used to approximate the number of linear regions of the functions represented by GNNs, SCNNs (Ebli et al., 2020) and
MPSNs.
We focus on the case where the message function is a linear layer and AGG is sum followed by ReLU.
We obtain new results in all cases, showing superior capacity of MPSNs.
The details of the notation and proofs of Theorems 9, 10, and 11 are relegated to Appendix C.
Theorem 9 (Number of linear regions of a GNN layer) .
Consider a graph G with S 0 nodes, node input features of dimension d ≥ 1 , and node output features of dimension m .
Suppose the aggregation function H as function of X is linear and invertible.
Then, the number of linear regions of the functions represented by a ReLU GNN layer (9) has the optimal upper bound R GNN = (cid:16) 2 (cid:80)
d − 1 i =0 (cid:0) m − 1 i (cid:1)(cid:17) S 0 .
This applies to aggregation functions with no trainable parameters including GCN convolution (Kipf & Welling, 2017), spectral GNN (Defferrard et al., 2016; Bruna et al., 2014), and traditional message passing (Gilmer et al., 2017).
The above result should be compared with the optimal upper bound for a standard dense ReLU layer without biases, which for d inputs and m outputs is 2 (cid:80) d − 1 i =
(cid:0) m − 1 i (cid:1) .
Theorem 10 (Number of linear regions for an SCNN layer) .
Consider a p -dimensional simplicial complex with S n n -simplicies for n = 0 , 1 , . .
p .
Suppose that each M n is invertible.
Then the number of linear regions of the functions represented by a ReLU SCNN layer (10) has the optimal upper bound R SCNN =
(cid:81) pn
=0 (cid:16) 2 (cid:80) d n − 1 i =0 (cid:0) m n − 1 i (cid:1)(cid:17) S n , where, for each of the n -simplices,
d n is the input feature dimension and m n is the number of output features.
Theorem 11 (Number of linear regions of an MPSN layer) .
With the above settings, the maximum number of linear regions of the functions represented by a ReLU MPSN layer is upper bounded by R MPSN ≤ (cid:81) pn =0 (cid:16) 2 (cid:80) d n − 1 + d n + d n +1 − 1 i = (cid:0) m − 1 i (cid:1)(cid:17) S n , where we set d − 1 =
d p +1 = 0 .
We also note the ‘trivial’ upper bound, with N := (cid:80) pn
=
S n d n and M := (cid:80) pn
=
S n m , R MPSN ≤ 2 (cid:80) N − 1 j =0 (cid:0) M − 1 j (cid:1) .
Moreover, if m ≤ d n and rank( O n ) =
S n for all n , writing S = max { S 0 , . . .
, S p } , we have the lower bound R MPSN ≥ 2 (cid:80) N − ( mS ) − 1 j = (cid:0) M − 1 j (cid:1) .
We note that the MPSN lower bound (26) surpasses the SCNN upper bound (22), with order (cid:81) n ( (cid:80) n (cid:48)
mS n (cid:48) )
d n S n and (cid:81) n ( mS n ) d n S n , respectively, when d n ≥ m .
The GNN bound (21) is a special case of the SCNN bound (22) with p = 0 , and has asymptotic order m d 0 S 0 .
{+Computational Complexity A d -simplex σ of a p -complex has d + 1 faces and there are (cid:0) d +12 (cid:1) upper adjacencies between its faces. Then, a message passing procedure relying on Theorem 1, which considers only these adjacencies, has a computational complexity Θ (cid:0) (cid:80) pd =
 ( d + 1) S d + (cid:0) d +12 (cid:1) S d (cid:1) = Θ (cid:0) (cid:80) pd =0 (cid:0)+}
d +12 (cid:1) S d (cid:1) .
If we consider p to be a small constant, which is common for many real-world datasets, then the binomial coefﬁcients can be absorbed in the bound, which results in a linear computational complexity in the size of the complex Θ( (cid:80) pd =0 S d ) . When operating on clique complexes, we also have to take into account the complexity of the preprocessing step. The number of k -cliques in a graph is upper-bounded by O ( n k ) and they can be listed in O ( a ( G ) k − 2 m ) time (Chiba & Nishizeki, 1985), where a ( G ) is the arboricity of the graph (i.e. a measure of graph sparsity) and m is the number of edges.
Since the arboricity can be shown to be at most O ( m 1 / 2 ) and m ≤ n 2 , all k -cliques can be listed in O ( n k − 2 m ) . In particular, all triangles can be found in O ( m 3 / 2 ) . For certain classes, such as planar graphs, where a ( G ) ≤ 3 , the complexity becomes linear in the size of the graph. Overall, the fact that the algorithm can take advantage of the sparsity of the graph makes it strictly better than the Ω( n k ) complexity of k -GNNs.
[Figure] Veriﬁcation on SR Graphs We experimentally validate our theoretical result on the expressive power of our proposed architecture on the task of distinguishing hard pairs of isomorphic graphs.
In particular, we benchmark our architecture on 9 synthetic datasets comprising families of
Strongly Regular graphs .
Strongly Regular (SR) graphs represent ‘hard’ instances of graph isomorphism, as pairs thereof cannot provably be distinguished by the 3 -WL test (we refer readers to Section B.1 for a formal proof).
Results are illustrated in Figure 3, where we show performance on our isomorphism problem in terms of failure rate, that is the fraction of non-distinguished pairs.
Our employed architecture is parameterised similarly to GIN (Xu et al., 2019b) and hence we refer to it as Simplicial
Isomorphism Network (SIN). As it can be observed, SIN is able to distinguish the majority of graph pairs in all families, since, in contrast to standard graph neural networks, it is able to access information related to the presence and number of cliques therein.
We additionally run an MLP model with sum readout on the same inputs.
SIN outperforms this strong baseline on certain families, showing the importance of performing simplicial message passing.
Finally, we run a GIN architecture comparable to SIN and empirically verify its inability to distinguish any pair, theoretically justiﬁed by its expressive power being upper-bounded by 1 -WL.
Additional experimental details are in Appendix G.1.
[Table]
Real-World Graph Classiﬁcation We study the practical impact of considering higher-order interactions via clique-complexes and report results for a few popular graph classiﬁcation tasks commonly used for benchmarking GNNs.
We follow the same experimental setting and evaluation procedure described in Xu et al.
(2019b). Accordingly, we report the best mean validation accuracy computed in a 10 -fold cross-validation fashion, as well the related standard deviation. Additional architectural details are in Appendix G.2. The performance of our SIN model are reported in Table 1, along with those of graph kernel methods and other GNNs.
SIN generally performs on-par with state-of-the-art approaches. We observe that our model has the highest mean performance on the IMDB datasets, which have the largest mean and median number of triangles. At the same time, on datasets like NCI1, where the number of higher-order structures is close to zero, the model shows the same mean accuracy as GIN.
Trajectory Classiﬁcation Here, we turn our attention to an application involving oriented simplicial complexes.
Inspired by Schaub et al.
(2020), we generate a synthetic dataset of trajectories on the simplicial complex from Figure 4.
All trajectories pass either through the bottom left corner or the top-right corner, thus giving rise to two different classes that we aim to distinguish.
Due to the two holes present in the complex, the trajectories of the two classes correspond to approximately orthogonal directions in the space of harmonic functions of the L 1 Hodge-Laplacian (Schaub et al., 2020).
Therefore, we hypothesise, that an orientation invariant MPSN network with orientation equivariant layers should easily distinguish the two classes.
We describe our model and other task details in Appendix G.3.
As a baseline, we consider a similar message passing GNN operating in the line graph of the 1-skeleton of the simplicial complex, without being (explicitly) aware of the orientation and the triangles present in the graph.
As expected, the MPSN model obtains 96.5% accuracy compared to 71.5% for the GNN baseline.
C ONCLUSION
We introduce a provably powerful message passing procedure for simplicial complexes relying on local higher-order interactions.
We motivate our message passing framework by the introduction of SWL, a colouring algorithm for simplicial complex isomorphism testing, generalising the WL test for graphs.
We prove that when graphs are lifted in the simplicial complex space via their clique complex, SWL and MPSNs are more expressive than the WL test.
At the same time, we produce an estimate for the number of linear regions of GNNs, SCNNs and MPSNs, which also reveals the superior expressive power of our model.
We empirically conﬁrm these results on a dataset of challenging strongly regular graphs, real-world graph classiﬁcation benchmarks, a trajectory classiﬁcation problem, and by computing 2D slices through the linear regions of the models. A CKNOWLEDGEMENTS
YW and GM acknowledge support from the European Research Council (ERC) under the EU’s
Horizon 2020 research and innovation programme (grant agreement n o 757983).
We would also like to thank Teodora Stoleru for creating Figure 1.
A B ACKGROUND
Simplicial complexes Combinatorially, a simplicial complex K is a family of sets closed under taking subsets (Hatcher, 2000).
A member σ ∈ K with cardinality k + 1 is called a k -dimensional simplex or simply a k -simplex.
One can interpret simplices as geometric objects: 0-simplices as vertices , 1-simplices as edges , 2-simplices as triangles , and so on.
A simplicial complex is intuitively a collection of such simplices that intersect ‘nicely’ (see Figure 1).
Each k -simplex has k + 1 faces of dimension k − 1 (e.g. the three edges of a triangle).
If σ 1 is a face of σ 2 , then σ 2 is a coface of σ 1 .
Two k -simplicies are lower adjacent if they have a common face (e.g. two triangles sharing an edge) and upper adjacent if they have a common coface (e.g. two vertices connected by an edge).
We denote by F ( σ ) , C ( σ ) , N ↓ ( σ ) , N ↑ ( σ ) the sets of faces, cofaces, lower and upper neighbours of simplex σ .
We use K d to refer to all the simplicies of dimension d in the complex.
An oriented simplicial complex is a simplicial complex together with a total order on its set of 0 -simplices.
The orientation of a simplicial complex is needed for deﬁning simplicial homology (Hatcher, 2000) – the theory of counting d -dimensional ‘holes’ in the complex (e.g. connected components, cycles or voids).
However, orientations do not encode any topological information, as simplicial homology is independent of the choice of order.
Graph neural networks GNNs are neural models for processing signals on 1-dimensional simplicial complexes (graphs).
Most GNN models follow the message passing scheme of Gilmer et al.
(2017) or its simpliﬁed variants.
Under this framework, the t th layer of a GNN performs the following message and updates computations:
[Equation]
where h tv denotes the features of node v at layer t , h te ( v,w ) represents the features of the edge e ( v, w ) and N ( v ) is the set of neighbours of node v .
Some architectures do not use any edge features.
AGG is a permutation-invariant aggregation operator such as sum, mean, or max, and the message and update functions M and U are typically learnable.
For node/edge regression and classiﬁcation tasks, the ﬁnal representations h Lv and h Le ( v,w ) are used to make the prediction.
For graph-level tasks, such as graph classiﬁcation, all the ﬁnal individual representations of the nodes are combined into a single vector by a readout function:
[Equation]
Most GNN architectures differ only in the design of aggregate (AGG), message ( M ) and update ( U ).
The Weisfeiler-Lehman (WL) test The WL (or 1-WL) graph-isomorphism test is a fast heuristic providing a necessary condition for two graphs to be isomorphic (Weisfeiler & Leman, 1968).
It is an iterative color reﬁnement procedure for the graph nodes.
Given a graph G , all the nodes v ∈ V G are initialised with the same colour c 0 v ; they are then iteratively reﬁned as: [Equation] }} u ∈N ( v ) (cid:1) , that is by applying an injective map on the multiset of neighboring colours.
The algorithm stops when the graph colouring is stable (colours are no longer updated), yielding a colour histogram.
If two graphs are associated with two different colour histograms they are deemed non-isomorphic.
However, if the histograms are identical, then the WL test is non-conclusive: the two graphs are possibly , but not necessarily , isomorphic.
The analogy between color reﬁnement and message passing has been analyzed in Xu et al.
(2019b)
; Morris et al.
(2019), where the expressive power of standard graph neural networks is at most as the WL test.
The k -WL test is a higher-order extension of the WL algorithm that works on k -tuples instead of individual nodes.
With the exception of 1-WL and 2-WL tests (that are equivalent), ( k + 1) -WL is strictly stronger than k -WL, for any k ≥ 2 , that is, there exist graphs on which k -WL fails and ( k + 1) -WL succeeds, but not vice versa.
k -WL is thus a hierarchy or increasingly more powerful graph isomorphism tests, sometimes referred to as the Weisfeiler-Lehman hierarchy .
Maron et al.
(2019) proposed a form of graph neural networks equivalent to 3 -WL that however lack locality and suffer from high memory and computational complexity.
B P ROOFS OF SWL T HEORY R ESULTS
We ﬁrst introduce the required notion and notation.
We note that even though these results mainly refer to simplicial complexes, they also apply to graphs because any graph is also a simplicial complex.
The ﬁrst deﬁnition formalises the notion of ‘colouring’ for the simplices of a simplicial complex.
Deﬁnition 12 (Colouring function) .
A colouring function is a function c : K → Σ that takes as input a simplex from a simplicial complex K and outputs a colouring of that simplex from a countable alphabet Σ .
We denote the space of all colouring functions for a complex K as C K .
We abuse the notation slightly and use c v interchangeably with c ( v ) .
Deﬁnition 13 (Histogram) .
The histogram of a multiset {{ σ i }} with a i ∈ Σ is a function H : Σ → N , where H ( σ ) speciﬁes the number of occurences of σ in the multiset.
The next deﬁnition generalised the concept of a colouring algorithm such as WL or SWL.
Deﬁnition 14 (Colouring algorithm) .
A colouring algorithm
A is a computable function that takes as input a complex K and outputs a colouring from C K with the property that for any two K 1 and K 2 , the colour histograms of A ( K 1 ) and A ( K 2 ) over the simplicies of a certain dimension are different only if K 1 and K 2 are not isomorphic.
In other words, if the colour histograms at any level of the complex (vertices, edges, triangles, etc.) are different, it implies that the simplicial complexes are different.
However, this does not mean any two non-isomorphic simplicial complexes must have different colourings.
Finally, we introduce the notion of colour-reﬁnement as a way of comparing two colourings.
Deﬁnition 15 (Simplicial complex colour reﬁnement) .
Let c 1 and c 2 be two colouring functions for a complex K .
We say that c 1 (cid:118) c 2 (i.e. c 1 color-reﬁnes c 2 ) if for any two simplicies v, w ∈ K we have that c 1 ( v ) =
c 1 ( w ) implies that c 2 ( v ) =
c 2 ( w ) .
We now prove a lemma that will be used repeatedly in our proofs in this section.
It shows how color reﬁnement can be used to compare the ability of certain coloring algorithms.
Lemma 16.
Let A 1 and A 2 be two colouring algorithms.
If A 1 ( K ) (cid:118)
A 2 ( K ) for any complex K , then any two non-isomorphic simplicial complexes K 1 and K 2 that can be distinguished by A 2 can also be distinguished by A 1 .
In other words, A 1 is at least as powerful as A 2 .
Proof.
Let K 1 and K 2 be two non-isomorphic simplicial complexes that can be distinguished by A 2 .
Because they can be distinguished, this means that there exists a certain dimension where the colour histograms produced by A 2 ( K 1 ) and A
( K 2 ) are different.
We denote these histograms by sets of tuples { ( c i , f i ) } i that contain
a unique colour c
i and its frequency f i among the simplicies of the given dimension.
The two sets can be different in three (potentially concurrent) ways:
A 2 ( K 1 ) contains a colour not present in A 2 ( K 2 )
(among the simplicies of the considered dimension).
A 2 ( K 2 ) contains a colour not present in A 2 ( K 1 ) (among the simplicies of the considered dimension).
The frequencies of a certain colour present in both histograms are different.
By A 1 ( K ) (cid:118)
A 2 ( K ) for any K , any colour histogram under A 1 will ‘reﬁne’ the colour histogram under A 2 .
That is, it splits each H 2 ( c i ) =
f i in the A 2 histogram ( H 2 ) into multiple sub-colours with their own frequency H 1 ( c i,j ) =
f i,j where j is an index over the sub-colours, H 2 is the histogram produced by the colouring of A 1 , and (cid:80) j H 1 ( c i,j ) =
H 2 ( c i ) .
It then implies that the histograms of the two graphs will also be different under the colouring of A 1 for any of the three cases listed above.
Let us now proceed by proving Theorem 1.
We will ﬁrst introduce and prove the following:
Lemma 17.
SWL with HASH (cid:0) c tv , c t F ( v ) , c t ↓ ( v ) ,
c t ↑ ( v ) (cid:1) is as powerful as SWL with the generalised update rule HASH (cid:0) c tv , c t F ( v ) , c t C ( v ) , c t ↓ ( v ) , c t ↑ ( v ) (cid:1) .
Proof.
Let a denote the colouring of the general version and b the colour of the restricted version.
Then, a (cid:118) b because of considering the additional cofaces in the colouring procedure.
We will now prove the converse.
Suppose b t +1 v =
b t +1 w .
Then, we have that the arguments of the hash function are equal.
Thus, b tv =
b tw , b t ↓ ( v ) = b t ↓ ( w ) , b t ↑ ( v ) =
b t ↑ ( w ) , and b t F ( v ) =
b t F ( w ) .
Because the multi-sets of the upper colours are equal, the multi-sets of the second entries of the tuples inside them (i.e. the co-face colours) are also equal to {{ b tv ∪ v (cid:48) | v (cid:48) ∈ N ↑ ( v ) }} = {{ b tw ∪ w (cid:48) | w (
cid:48) ∈ N ↑ ( w ) }} .
As v and w have the same dimension and their multi-sets have the same size, they have the same number of cofaces.
Due to that each of these co-faces shows up the same number of times in the multi-sets, the colours of the cofaces are thus the same: {{ b tx | x ∈ C ( v ) }} = {{ b ty | y ∈ C ( w ) }} .
By the induction, the arguments of the hash function used by a are the same for v and w .
Thus, a t +1 v = a t +1 w .
Proof of Theorem 1.
Let b denote the colouring based on the ﬁrst rule and a the colouring as used in the proof of Lemma 17.
As before, it is trivial to show a t (cid:118) b t by the additional argument (the colours of the down adjacencies).
We now prove that b 2 t (cid:118) a t by induction.
The base case holds by deﬁnition.
We assume the statement is true for t and prove it for t + 1 , as follows.
Suppose b 2 t +2 v = b 2 t +2 w .
By expanding the hash function two steps back in time, we obtain b 2 tv =
b 2 tw , b 2 t F ( v ) =
b 2 t F ( w ) , b 2 t ↑ ( v ) =
b 2 t ↑ ( w ) .
Suppose for the sake of contradiction that b 2 t ↓ ( v ) (cid:54) =
b 2 t ↓ ( w ) .
This means that there exists a pair of colours ( l 0 = b 2 tx , l 1 = b 2 tx ∩ v ) with x ∈ N ↓ ( v ) that shows up (without loss of generality) more times in b 2 t ↓ ( v ) than in b 2 t ↓ ( w ) .
Let us partition these apparitions in these two multi-sets by the faces f 1 ∈ F ( v ) and f 2 ∈ F ( w ) having the colour b 2 tf 1 = b 2 tf 2 = l 1 .
By b 2 t F ( v ) =
b 2 t F ( w ) , we have the same number of such faces / partitions for v and w .
Note that for each such face f of v , the size of the set
A v ( f ) = { x | b 2 tx = l 0 , x ∈ N ↓ ( v ) , f = x ∩ v } (containing the lower adjacent simplices of v with colour l 0 and sharing face f ) determines the colour of f at step t
+ 1 since the colours {{ b 2 tx | x ∈
A v ( f ) }} show up in the tuples of b 2 t ↑ ( f ) , which is an input of the hash function colouring f .
The same applies for the faces of w .
Because there are more colors l
= ( l 0 , l 1 ) in b 2 t ↓ ( v ) than in b 2 t ↓ ( w ) (by the assumption in the paragraph above), the histogram of the sizes of these partitions are different between v and w .
This means that in the next time step, {{ b 2 t +1 f}} (cid:54) =
{{ b 2 t +1 f}} .
Since these colours are also different from the colours of the other faces of v and w at step 2 t
+ 1 , we have that b 2 t +1 F ( v ) (cid:54)
[Equation]
b [Equation] ( w ) .
Finally, this implies that b 2 t +2 v (cid:54) =
c 2 t +2 w , which is a contradiction.
Therefore, b 2 t ↓ ( v ) =
b 2 t ↓ ( w ) .
Now, we apply the induction hypothesis to obtain that a t v = a t w , a t F ( v ) =
a t F ( w ) , a t ↑ ( v ) = a t ↑ ( w ) and a t ↓ ( v ) = a t ↓ ( w ) .
Then, b 2 t (cid:118) a t .
We show a slightly weaker version of Theorem 2.
Lemma 18.
SWL is at least as powerful as WL.
Proof.
Let G be a graph with a clique complex K .
Let c t be the colouring of the nodes of G at iteration t of WL and sc t the colouring of the (same) vertices in K at iteration t of SWL.
To prove the lemma, we will show by induction that sc t is a reﬁnement of c t .
Combined with Lemma 16, this proves the result.
For the base case, the implication holds at initialisation since all nodes are assigned the same colour.
That is: sc 0 ( v ) =
sc 0 ( w ) =
⇒ c 0 ( v ) =
c 0 ( w ) .
For the induction step, suppose sc t +1 ( v ) =
sc t +1 ( w ) , for two 0 -simplices v and w in K .
In this case, the arguments of the HASH function must be equal.
Concretely, as vertices are only upper adjacent and have no faces, sc t ( v ) =
sc t ( w ) and sc t ↑ ( v ) =
sc t ↑ ( w ) .
In turn, the sets formed of the ﬁrst entry of the tuples in these two multi-sets are also equal: SC v = {{ sc tx }} = {{ sc ty }}
= SC w , for all x ∈ N ↑ ( v ) and y ∈ N ↑ ( w ) .
By the induction hypothesis, the equalities sc t ( v ) =
sc t ( w ) and SC v = SC w imply c t ( v ) =
c t ( w ) and
C v = {{ c tx }} = {{ c ty }} =
C w .
Since these are the arguments the WL hash function uses to compute the colours of v and w at the next step, we obtain c t +1 ( v ) =
c t +1 ( w ) .
Informally, this proof shows that the information coming from the colouring of the upper layers of the complex will reﬁne the colouring of the vertices.
This means that SWL will be able to distinguish just through its vertex-level colour histogram at least the same set of graphs that WL can distinguish.
However, this proof disregards the histograms of the higher-levels and these can indeed be used to show that SWL is strictly more powerful than WL, which is done in Theorem 2.
Proof of Theorem 2.
Based on Lemma 18, it is sufﬁcient to present a pair of graphs that cannot be distinguished by WL, but whose clique complexes can be distinguished by SWL.
Such a pair is included in Figure 5.
While WL produces the same colouring for both graphs, one clique complex contains two triangles, while the other has no triangles.
Proof of Lemma 4.
Let sc tv and h tv be the colouring of the simplex v at iteration t of SWL and the t -th layer of an MPSN, respectively.
We will show by induction that sc t reﬁnes the colouring of h t .
For this proof, it is convenient to use the most general version of SWL, containing the complete set of adjacencies.
As above, the base case holds since both methods start with the same initial colour.
For the induction step, suppose we have two simplicies v and w such that sc t +1 ( v ) =
sc t +1 ( w ) .
Because the SWL colouring is an injective mapping, the arguments to the HASH function are the same.
This means that sc t ( v ) =
sc t ( w ) and the multi-sets of colours formed by their neighbours are identical: sc t ↓ ( v ) =
sc t ↓ ( w ) , sc t ↑ ( v ) =
sc t ↑ ( w ) , sc t F ( v ) =
sc t F ( w ) , sc t C ( v ) =
sc t C ( w ) .
By the induction hypothesis, these multi-sets will be equal under the colouring h t .
Enumerating all, h t ( v ) =
h t ( w ) , h t ↓ ( v ) = h t ↓ ( w ) , h t ↑ ( v ) =
h t ↑ ( w ) , h t F ( v ) =
h t F ( w ) , h t C ( v ) =
h t C ( w ) .
Because the exact same multi-sets are supplied as input to the MESSAGE, AGGREGATE and UPDATE functions, their output are also the same for v and v .
Then, h t +1 v = h t +1 w .
Applying Lemma 16 thus proves the lemma.
Proof of Theorem 5.
By Lemma 4, we only need to show that for an MPSN model satisfying the conditions in the theorem, we have that h t (cid:118) sc t .
The base case can be proved by deﬁnition.
For the step case, given that the UPDATE, AGGREGATE and MESSAGE functions are injective, for any two simplicies h t +1 v = h t +1 w , the inputs to these functions at the previous iteration were also the same.
As in our previous proofs, by applying the induction hypothesis, the inputs to the SWL HASH function at iteration t for v and w are also equal and sc t +1 v =
sc t +1 w .
It follows h t (cid:118) sc t , sc t (cid:118) h t (Lemma 4) and, consequently, sc t = h t .
B. H IGHER - ORDER WL AND S TRONGLY R EGULAR G RAPHS
Higher-order variants of the standards WL procedure operate on node tuples rather than single nodes and iteratively apply color reﬁnement steps thereon.
k -WL
The k -WL is one such higher-order variants.
It speciﬁcally operates on node ktuples by reﬁning their colors based on the generalized notion of j -neighborhood.
The jneighborhood ( j ∈ { 1 , . . . , k } ) for node k -tuple v = ( v 1 , v 2 , . .
. , v k ) is deﬁned as N j ( v ) =
{ ( v 1 , . .
. , v j − 1 , w, v j +1 , . .
, v k ) | w ∈ V G } .
The algorithm ﬁrst initialises node tuples based on their isomorphism type: two k -tuples v a =
( v a 1 , v a 2 , . .
. , v ak ) , v b = ( v b 1 , v b 2 , . . .
, v bk ) have the same isomorphism type (and are thus assigned the same initial colour c v a = c v b ) iff (i) ∀ i, j ∈ { 1 , . .
. , k } , v a i = v a j ⇔ v b i = v b j , (ii) ∀ i, j ∈ { 1 , . . . , k } , v a i ∼ v a j ⇔ v b i ∼ v b j , where ∼ indicates adjacency.
Given this initial colouring, the procedure iteratively applies the following color reﬁnement step
[Equation]
until the colouring does not change further.
The k -WL procedure can be employed to test the isomorphism between graphs in the same way as the standard WL one is.
For any k ≥ 2 , it is known that ( k + 1) -WL test is strictly stronger than k -WL one, i.e. there exist exemplary pairs of non-isomorphic graphs that k -WL cannot distinguish while ( k +1 )
-WL can, but not vice-versa.
Local variants of k -WL have recently been introduced in Morris et al.
k -FWL
The k -Folklore WL procedure ( k -FWL) is another higher-order variant of the standard WL.
Similarly to k -WL, it operates by reﬁning the colors of node k -tuples, initialised based on their isomorphism type.
However, it employs a different notion of neighborhood and reﬁnement step.
The Folklore j -neighborhood for node k -tuple v is deﬁned as N Fj ( v ) =
(cid:0) ( j, v 2 , . . .
, v k ) , ( v 1 , j, . .
, v k ) , . . , ( v 1 , . .
. , v k − 1 , j ) (cid:1) , with j ∈ V G .
The algorithm iteratively applies the steps
[Equation]
until the colouring does not change.
It is known that k -FWL is equivalent to ( k + 1 )-WL for k ≥ 2 .
Strongly Regular Graphs A Strongly Regular graph in the family SR( n , d , λ , µ ) is a regular graph with n nodes and degree d , for which every two adjacent nodes always have λ mutual neighbours and every two non-adjacent nodes always have µ mutual neighbours.
This class of graphs is of particular interest due to the following lemma.
Lemma 19.
No pair of Strongly Regular graphs in family SR( n , d , λ , µ ) can be distinguished by the 2 -FWL test.
Proof.
Let us denote by V 2 G the set of all node 2 -tuples in graph G .
We note that three isomorphism types are induced by considering node 2 -tuples:
(1) node type : v = ( v 1 , v 1 ) (2) edge type : v = ( v 1 , v 2 ) with v 1 ∼ v 2 (the two nodes are adjacent in the original graph) (3) non-edge type : v = ( v 1 , v 2 ) with v 1 (cid:54)∼ v 2 (the two nodes are not adjacent in the original graph).
These three isomorphism types partition the tuple set V 2 G into the three subsets V 2 G (1) , V 2 G (2) , V 2 G (3) such that any tuple v ∈ V 2 G ( i ) is of isomorphism type i .
We write v ( i ) to indicate v ∈ V 2 G ( i ) for simplicity.
At initialisation, the 2 -FWL algorithm assigns a colour to each tuple v ∈ V 2 G based on its isomorphism type, that is ∀ v ∈ V 2 G ( i ) , c v = c
i .
The colouring is therefore constant within partitions.
Then, we notice that the colouring is kept constant within partitions through the application of the reﬁnement steps described by Equation 7.
In other words, the 2 -FWL procedure cannot produce a colour partitioning of the set of node 2 -tuples that is ﬁner than the one at initialisation.
This is shown by induction on the step t of color reﬁnement.
The base case evidently holds for t = 0 since all tuples in the same partition are assigned the same colour by the 2 -FWL initialisation procedure.
For the induction step we assume that the colouring is constant within each partition at t and show that it maintains constant at t + 1 , that is, after the application of one colour reﬁnement step.
This is proved by showing that all node tuples within the same partition have their colour reﬁned identically.
We will show this for each of the three partitions separately, leveraging on the induction hypothesis and the properties of Strongly Regular graphs.
A node tuple v = ( v 1 , v 1 ) ∈ V 2 G (1) has N Fj ( v )
= (cid:0) ( j, v 1 ) , ( v 1 , j ) (cid:1) , j ∈ V G .
Therefore, any v ∈ V 2 G (1) has exactly:
• ( j = v 1 ) 1 neighborhood of the form (cid:0) w (1) , w (1) (cid:1) , associated with color tuple ( c t 1 , c t 1 ) ; • ( j ∼ v 1 ) d neighborhoods of the form (cid:0) w (2) , u (2) (cid:1) , associated with color tuple ( c t 2 , c t 2 ) ; • ( j (cid:54)∼ v 1 ) n − d − 1 neighborhoods of the form (cid:0) w (3) , u (3) (cid:1) associated with color
tuple
( c t 3 , c
t For any v ∈ V 2 G (1) we then have
[Equation]
d [Equation] times }} . A node tuple v = ( v 1 , v 2 ) ∈ V 2 G (2) has N Fj ( v )
= (cid:0) ( j, v 2 ) , ( v 1 , j ) (cid:1) , j ∈ V G .
Therefore, any v ∈ V 2 G (2) has exactly:
• ( j = v 2 ) 1 neighborhood of the form (cid:0) w (1) , u (2) (cid:1) , associated with color tuple ( c t 1 , c t 2 ) ; • ( j = v 1 ) 1 neighborhood of the form (cid:0) w (2) , u (1) (cid:1) , associated with color tuple ( c t 2 , c t 1 ) ; • ( j ∼ v 2 , j ∼ v 1 ) λ neighborhoods of the form (cid:0) w (2) , u (2) (cid:1) , associated with color tuple ( c t 2 , c t 2 ) ; • ( j ∼ v 2 , j (cid:54)∼ v 1 ) d − λ neighborhoods of the form (cid:0) w (2) , u (3) (cid:1) , associated with color tuple ( c t 2 , c t 3 ) ; • ( j (cid:54)∼ v 2 , j ∼ v 1 ) d − λ neighborhoods of the form (cid:0) w (3) , u (2) (cid:1) , associated with color tuple ( c t 3 , c t 2 ) ; • ( j (cid:54)∼ v 2 , j (cid:54)∼ v 1 ) k
= n − 2 − 2 d + λ neighborhoods of the form (cid:0) w (3) , u (3) (cid:1) , associated with color tuple ( c t 3 , c t 3 ) .
For any v ∈ V 2 G (2) we have c t +1 v = HASH (cid:16) c
t 2 , M F,t ( v ) (cid:17)
M F,t ( v ) = {{ ( c t 1 , c t 2 ) (cid:124) (cid:123)(cid:122) (cid:125) 1 time , ( c t 2 , c t 1 ) (cid:124) (cid:123)(cid:122) (cid:125) 1 time , ( c t 2 , c t 2 ) (cid:124) (cid:123)(cid:122) (cid:125)
λ times , ( c t 2 , c t 3 ) (cid:124) (cid:123)(cid:122) (cid:125)
d − λ times , ( c t 3 , c t 2 ) (cid:124) (cid:123)(cid:122) (cid:125)
d − λ times , ( c t 3 , c t 3 ))
(cid:124) (cid:123)(cid:122) (cid:125) k times }} .
A node tuple v = ( v 1 , v 2 ) ∈ V 2 G (3) has N Fj ( v )
= (cid:0) ( j, v 2 ) , ( v 1 , j ) (cid:1) , j ∈ V G .
Therefore, any v ∈ V 2 G (3) has exactly:
• ( j = v 2 ) 1 neighborhood of the form (cid:0) w (1) , u (3) (cid:1) , associated with color tuple ( c t 1 , c t 3 ) ; • ( j = v 1 ) 1 neighborhood of the form (cid:0) w (3) , u (1) (cid:1) , associated with color tuple ( c t 3 , c t 1 ) ; • ( j ∼ v 2 , j ∼ v 1 ) µ neighborhoods of the form (cid:0) w (2) , u (2) (cid:1) , associated with color tuple ( c t 2 , c t 2 ) ; • ( j ∼ v 2 , j (cid:54)∼ v 1 ) d − µ neighborhoods of the form (cid:0) w (2) , u (3) (cid:1) , associated with color tuple ( c t 2 , c t 3 ) ; • ( j (cid:54)∼ v 2 , j ∼ v 1 ) d − µ neighborhoods of the form (cid:0) w (3) , u (2) (cid:1) , associated with color tuple ( c t 3 , c t 2 ) ; • ( j (cid:54)∼ v 2 , j (cid:54)∼ v 1 ) k
= n − 2 − 2 d + µ neighborhoods of the form (cid:0) w (3) , u (3) (cid:1) , associated with color tuple ( c t 3 , c t 3 ) .
For any v ∈ V 2 G (3) we then have
[Equation]
(cid:124) (cid:123)(cid:122) (cid:125) 1 time , ( c t
[Equation]
c t 1 ) (cid:124) (cid:123)(cid:122) (cid:125) 1 time , ( c t 2 , c t 2 ) (cid:124) (cid:123)(cid:122) (cid:125) µ times , ( c t 2 , c t 3 ) (cid:124) (cid:123)(cid:122) (cid:125)
d − µ times , ( c t 3 , c t 2 ) (cid:124) (cid:123)(cid:122) (cid:125)
d − µ times , ( c t 3 , c t 3 )) (cid:124) (cid:123)(cid:122) (cid:125) k times }} .
This proves the induction and conﬁrms that all tuples in the same partition have the same colour at any colour reﬁnement time step t .
If the colouring is constant within partitions at any 2 -FWL step, then the colour histogram associated with a graph at step t purely depends on the cardinality of each of the three.
We notice that, for any
G ∈ SR( n , d , λ , µ ) , they are completely determined by the ﬁrst two parameters:
[Equation]
Given the above, any two graphs G 1 , G 2 ∈ SR( n , d , λ , µ ) are associated with the same colour histograms at any step of the 2 -FWL procedure and, therefore, cannot possibly deemed non-isomorphic by the last.
We leverage on Lemma 19 to prove Theorem 3.
Proof of Theorem 3.
In virtue of Lemma 19 and the fact that 2 -FWL is as powerful as 3 -WL, Theorem 3 is proved by exhibiting a pair of Strongly Regular graphs in the same family that are distinguished by the SWL test.
This pair is given by the two graphs in Figure 6: Rook’s 4 x 4 graph ( G 1 ) and the Shrikhande graph ( G 2 ), (the only) members of the SR(16,6,2,2) family of Strongly
Regular graphs.
The SWL test which considers their clique 3 -complexes distinguish them due to the fact that, differently from G 1 , G 2 possesses no 4 -cliques, thus its associated complex has no 3 -simplices.
C L INEAR R EGIONS OF GNN, SCNN AND MPSN
While the WL test has been used for studying the expressive power of GNNs, other tools have been used to study the expressive power of conventional neural networks, like fully connected and convolutional.
One such tool is based on the number of linear regions of networks using piece-wise linear activations (Pascanu et al., 2013; Mont´ufar et al., 2014).
This number has been used to draw distinctions between the expressive power of shallow and deep network architectures (Pascanu et al., 2013; Mont´ufar et al., 2014).
It can also be related to the approximation properties of the networks (Telgarsky, 2016) and it has also been considered to shed light into the representational power of (standard) convolutional networks (Xiong et al., 2020).
We show how this tool can also be used to approximate the number of linear regions of the functions represented by GNNs and message passing simplicial networks.
We focus on the case where the message function is a linear layer and AGG is sum followed by ReLU.
We obtain new results in all cases, showing superior capacity of MPSNs.
A network with piece-wise linear gates computes functions that are piece-wise linear over the input space.
A linear region is a maximal connected subset of the input over which the function is afﬁne.
Background on hyperplane arrangements A function f : R N → R M is a piecewise linear function if its graph { ( x, f ( x )): x ∈ R N } ⊆ R N × R M consists of a ﬁnite number of polyhedral pieces.
Projecting these polyhedra back onto R N by ( x, y )
(cid:55)→ x deﬁnes a polyhedral subdivision of R N .
The linear regions of the function are the N dimensional pieces in this subdivision.
These are the (inclusion maximal) connected regions of the input space where the function is afﬁne linear.
Let ψ : R → R ; s (cid:55)→ max { 0 , s } be the linear rectiﬁcation.
A ReLU with N inputs deﬁnes a function y : x (cid:55)→ ψ ( w (cid:62) x ) , which for any ﬁxed value of the weight vector w ∈ R N , w (cid:54) = 0 , has gradient with respect to the input vector x ∈ R N equal to 0 on the open halfspace { x : w (cid:62) x < 0 } and equal to w on the open halfspace { x : w (cid:62) x > 0 } .
Hence a ReLU deﬁnes a piecewise linear function with two linear regions.
A layer of ReLUs ψ ( w (cid:62) i x ) , i = 1 , . .
, M has linear regions given by the intersection of linear regions of the individual ReLUs.
The number of linear regions of the function represented by the layer is equal to the number of connected components that are left behind once we remove ∪ M i =
1 A i from R N , where A i = { x ∈ R N : w (cid:62) i x = 0 } is the hyperplane dividing the two linear regions of the i th ReLU.
Hence the linear regions of a layer of ReLUs can be described in terms of a hyperplane arrangement , i.e. a collection A
= { A i : i = 1 , . . .
, M } of hyperplanes.
An arrangement of hyperplanes in R N is in general position if the intersection of any k hyperplanes in the arrangement has the expected dimension.
We will focus on central arrangements, where each hyperplane contains the origin.
A central arrangement is in general position when the normal vectors w i 1 , . . .
, w i k of any k ≤ N of the hyperplanes are linearly independent.
The following well-known result from the theory of hyperplane arrangements will be particularly important in our discussion.
Theorem 20.
Let A be a central arrangement of M hyperplanes in R N in general position.
Then the number of regions of the arrangement, denoted r ( A ) , is equal to 2 (cid:80) N − 1 j =0 (cid:0) M − 1 j (cid:1) .
This is also the maximum number of regions of any central arrangement of M hyperplanes in R N .
This result can be derived from Zaslavsky’s theorem (Zaslavsky, 1975), which expresses the number of regions of an arbitrary arrangement, not necessarily in general position, in terms of properties of a partially ordered set, namely the collection of intersections of the hyperplanes in the arrangement partially ordered by reverse inclusion.
We will focus on central arrangements, but we point out that similar results to Theorem 20 can be derived for the case of non-central hyperplane arrangements.
An non-central arrangement of (afﬁne) hyperplanes in R N is in general position when any k ≤ N of the hyperplanes intersect in a set of dimension N − k , and any k > N of the hyperplanes have an empty intersection.
For such an arrangement, the number of regions is (cid:80) Nj =0 (cid:0) Mj (cid:1) .
The main challenges in computing the number of regions deﬁned hyperplane arrangements happen when the hyperplanes satisfy some type of constraints and are not in general position.
The type of layers that we discuss in the following correspond to central arrangements subject to certain constraints, namely that the normal vectors are the rows of a matrix with a particular block Kronecker product structure.
GNN The result in Theorem 11 contrasts with the following complexity of the plain GNNs.
We start with the simple case of Graph Neural Networks (GNNs).
A graph G =
( V, E, ω ) is a set of triplets with vertices
V = { v i } S 0 i =1 , edges E ⊆ V × V , and edge weight function ω : E → R .
The graph has an adjacency matrix A with the ( i, j ) th entry a ij = ω ( v i , v j ) .
Each node has a d -dimensional feature, and we collect the feature vectors into a matrix X ∈ R S 0 × d .
We consider a GNN convolutional layer of the form
[Equation]
where ψ is the entry-wise ReLU,
H ( A, X ) is an aggregation mapping, and W 0 ∈ R d
× m are the trainable weights.
We restate in the following theorem for Theorem 9 and give at full proof.
Theorem 21 (Number of linear regions of a GNN layer) .
Consider a graph G with S 0 nodes, node input features of dimension d ≥ 1 , and node output features of dimension m .
Suppose the aggregation function H as function of X is linear and invertible.
Then, the number of linear regions of the functions represented by a ReLU GNN layer (9) has the optimal upper bound
[Equation]
This applies to aggregation functions with no trainable parameters including GCN convolution (Kipf & Welling, 2017), spectral GNN (Defferrard et al., 2016; Bruna et al., 2014), and traditional message passing (Gilmer et al., 2017).
Proof.
For simplicity, we write Y = H
( A, X ) T ∈ R d × S 0 and V = W T ∈
R m × d .
We denote
Y :
j the j the column of Y , and V i : the i th row of V .
The GNN layer deﬁnes hyperplanes, for i = 1 , . .
, m, j = [Equation]
, S 0 ,
[Equation]
The arrangement A
= { A ij : i = 1 , . . .
, m, j = 1 , . .
. , S 0 } is a direct sum of the arrangements
A j = { A ij : i = 1 , . . .
, m } , j = 1 , . .
, S 0 .
It can be shown (see Zaslavsky, 1975) that this implies r ( A ) =
(cid:81) S
j =1 r ( A j ) .
Each A j is an arrangement of m hyperplanes in R N , N = dS 0 , whose normals span a subspace of dimension at most d , irrespective of S 0 .
Counting the number of regions deﬁned by A j is equivalent to counting the number of regions deﬁned by its essentialization
ess
( A j ) , which is the arrangement that the hyperplanes deﬁne on the span of their normal vectors.
We can regard ess
( A j ) as a (central) arrangement of m hyperplanes in R d with normals
V i : ∈ R d , i = 1 , . . .
, m .
For generic choices of the weight matrix W (cid:62)
=
V , this is a central arrangement in general position.
Hence, by Theorem 20,
[Equation]
For the number of regions of the entire arrangement A , corresponding to the number of linear regions of the function expressed by the layer, we obtain R GNN = r ( A ) =
(cid:81) S
j =1 r ( A j ) = (cid:16) 2 (cid:80) d − 1 i =0 (cid:0) m − 1 i (cid:1)(cid:17) S 0 .
This concludes the proof.
The above result should be compared with the optimal upper bound for a standard dense ReLU layer without biases, ψ ( Wx ) , which for d inputs and m outputs is 2 (cid:80) d − 1 i =
(cid:0) m − 1 i (cid:1) .
The invertibility condition for the aggregation function H can be relaxed, but is satisﬁed by many commonly used graph convolutions: i) For an undirected graph, the normalised adjacency matrix has non-negative eigenvalues.
If the eigenvalues are all positive, the aggregation function is invertible.
ii)
The Fourier transform is the square matrix of eigenvectors, as used in spectral GNN (Bruna et al., 2014).
When the graph Laplacian is non-singular, Fourier transform matrix, that is aggregation mapping is invertible.
iii)
For the transform Φ by graph wavelet basis, Haar wavelet basis or graph framelets, the Φ are all invertible (Xu et al., 2019a;
Li et al., 2020; Zheng et al., 2020a;b; 2021;
Wang et al., 2020).
So the bound in Theorem 21 applies to them.
SCNN Simplicial Complex Neural Networks (SCNNs) were proposed by Ebli et al.
An
SCNN layer for features on a p -simplicial complex is deﬁned in terms of matrices M n (e.g. simplicial
Laplacian) by
[Equation]
In this type of layer, the features on simplices of different dimensions n = 0 , 1 . .
. , p are computed in parallel.
The following theorem is a restatement of Theorem 10, for which we now give full proof and explanation.
Theorem 22 (Number of linear regions for an SCNN layer) .
Consider a p -dimensional simplicial complex with S n n -simplicies for n = 0 , 1 , . .
p .
Suppose that each simplicial Laplacian in dimension n L n is invertible.
Then the maximum number of linear regions of the functions represented by a ReLU SCNN layer N of the form (10) is
[Equation]
where, for each of the n -simplices, d n is the input feature dimension and m n is the number of output features.
Consider a p -dimensional simplicial complex with S n n -simplicies for n = 0 , 1 , . .
. , p .
Suppose that each simplicial Laplacian in dimension n L n M n is invertible.
Then the number of linear regions of the functions represented by a ReLU SCNN layer (10) of the form H out n
[Equation]
ψ ( L n H in n W n ) , n = 1 , . .
. , p , has the optimal upper bound
[Equation]
where, for each n -simplex, d n is the input feature dimension and m n is the number of output features.
The product over n in (22) reﬂects the fact that the features over simplices of different dimensions do not interact.
The GNN bound in Theorem 21 is recovered as the special case of the SCNN bound with p = 0 .
It is instructive to compare the SCNN bound in Theorem 22 with the optimal bound for a dense layer.
By Roth’s lemma (Roth, 1934), vec( M n H in n W n ) =
( W (cid:62) n ⊗ M n ) · vec( H in n ) , where vec denotes column-by-column vectorization and ⊗ the Kronecker product.
Hence, for each n , we can regard the
SCNN layer as a standard layer ψ ( Ux ) with weight matrix U =
( W (cid:62) n ⊗ M n ) ∈ R ( m n S n ) × ( S n d n ) and input vector x = vec( H in n ) ∈ R S n d n .
Notice that for the SCNN layer, the weight matrix has a speciﬁc structure.
A standard dense layer with S n d n inputs and m n S n ReLUs with generic weights and no biases computes functions with [Equation]
Proof of Theorem 22.
By the deﬁnition of the SCNN layer, for each dimension n , each of the S n n -simplices in the simplicial complex has d n input features.
Similar to the proof of Theorem 21, the arrangement for the n -dimensional simplices corresponds to a direct sum of S n arrangements, each of m n hyperplanes in R d n .
Hence the number of linear regions for this part of the complex is
[Equation]
Now for the entire layer, the arrangements for the different n are also combined as a direct sum, so that their number of regions multiply.
We have n ranging from dimension 0 to p , so that
[Equation]
This concludes the proof.
MPSN
In our Message Passing Simplicial Network (MPSN), the features on simplices of different dimensions are allowed to interact.
For a p -dimensional complex, denote the set of n -simplices by
S n with S n = |S n | .
Denote the n -simplex input feature dimension by d n , and the output feature dimension by m n = m , n = 0 , . . .
, p .
We consider an MPSN layer with linear message functions, sum aggregation for all messages and an update function taking the sum of the messages followed by a ReLU activation.
For each v ∈ S n , the output feature vector takes the form:
[Equation]
where ψ is an entry-wise activation ( s (cid:55)→ max { 0 , s } for ReLU), W n ∈ R d n × m n are trainable weight matrices, F ( v ) ⊆
S n − 1 denotes the set of ( n − 1) -faces and C ( v ) ⊆
S n +1 the set of ( n +1) -co-faces (containing faces) of v ∈ S n in the simplicial complex.
Further, U n ∈ R S n × S n − 1 , M n ∈ R S n × S n and O n ∈ R S n × S n +1 are some choice of adjacency matrices for the simplicial complex.
These could be, for instance, the Hodge Laplacian matrix L n and the corresponding boundary matrices B n , B (cid:62) n + that relate simplicies to their faces and cofaces (Barbarossa & Sardellitti, 2020).
We can rewrite (14) more generality and more concisely as follows.
For each n the output features on
S n can be written as
[Equation]
[Equation]
for some ﬁxed matrices U n ∈ R S n × S n − 1 , M n ∈ R S n × S n and O n ∈ R S n × S n +1 depending only on the simplicial complex.
To avoid clutter, we omit the superscript “in” of the input feature matrices
H n .
Concatenating (15) for all n , we can write the entire MPSN layer as
[Equation]
We will use this representation (or rather (15)) in the proof of the ﬁrst bound in Theorem 26 (which is a restatement of Theorem 11).
It is also useful to write the linear function in standard form.
Using Roth’s lemma, we can write (15) as
[Equation] [Equation]
Now, concatenating over n yields the expression ψ ( WH ) from (18) for the entire layer, we can write (14) as (details in Appendix C)
[Equation]
18) where H in = vec([ H in0 | H in1 | · · · | H in p ])
[Equation]
N , N
[Equation] [Equation]
vec([ H out0 | H out1 | · · · | H out p ]) ∈ R M with M = (cid:80) pn
=
S n m , and
W =
[Equation]
We study the number of linear regions of the function (18) with ReLU based on the matrix W ∈ R M × N .
For each of the output coordinates i ∈ { 1 , . . .
, M } , the ReLU splits the input space R N into two regions separated by a hyperplane { H in ∈ R N : W i : H in = 0 } with normal W (cid:62) i : ∈ R N .
In order to count the total number of regions, we will use results from the theory of hyperplane arrangements.
Zaslavsky (1975, Theorem A) shows that the number of regions r ( A ) deﬁned by an arrangement A of hyperplanes in R N is
[Equation]
where χ A is the characteristic polynomial of the arrangement.
By virtue of a theorem of Whitney (see Stanley 2004, Theorem 2.4 and Orlik & Terao 1992, Lemma 2.55), it can be written as χ A ( t ) =
(cid:80) ( − 1) |B| t N − rank( B ) , where the sum runs over subarrangements B ⊆
A that are central (hyperplanes in B have a nonempty intersection), and rank( B ) denotes the dimension spanned by the normals to the hyperplanes in B .
In our case, A is a central arrangement with normals given by the rows of the matrix W in (19).
Hence:
Lemma 23.
The number of linear regions of the function (18) with W ∈ R M × N and ψ being ReLU is equal to
[Equation]
where W B : denotes the submatrix of rows i ∈ B .
This formula counts the linear regions of any particular function represented by our layer.
Some interesting cases can be computed explicitly.
Proposition 24.
Consider some K ≤ N .
If rank( W B : ) =
min {| B | , K } for any B , then r ( A ) = [Equation]
Proof.
This result is known in theory of partial orders and hyperplane arrangements.
We include a proof which illustrates the evaluation of the characteristic polynomial.
If there is K so that rank( W B : )
= min {| B | , K } for all B , then Lemma 23 can be evaluated as
[Equation]
which is what was claimed.
We can characterize cases where the hypothesis of Proposition 24 is satisﬁed for particular values of
K .
For instance, using the tridiagonal block structure of W in (19):
Proposition 25.
If m ≤ d n and rank( O n ) = S n for n = 0 , . . .
, p , then rank( W B : ) ≥ min {| B | , K } , where K = N − m · max { S 0 , . . .
, S p } .
Proof.
Since matrix W from (19) is lower block triangular, we have
[Equation]
Since the matrix W is also upper block triangular, we have a similar bound in terms of the U n .
Using the rank formula for Kronecker products rank( W (cid:62) n ⊗ M n ) = rank( W (cid:62) n ) rank( M n ) , we see that if all the factor matrices are full rank, then rank( W B : ) ≥ (cid:80) p − 1 n =0 min { d n , m } S n for any
B ⊆ { 1 , . . .
, (cid:80) p − 1 n =0 mS n } .
Recall that W B : denotes the submatrix of W composed of the rows i ∈ B .
With the above proposition, we would obtain the following bounds for counting the linear regions of an MPSN layer.
This is a restatement of Theorem 11.
Here we give more details of the meaning and proof of the bounds.
Theorem 26 (Number of linear regions of an MPSN layer) .
With the above settings, the maximum number of linear regions of the functions represented by a ReLU MPSN layer (18) is upper bounded by
[Equation]
where we set d − 1 = d p +1 = 0 .
We also note the ‘trivial’ upper bound, with N := (cid:80) [-pn =
-]p n =0 S n d n
and
M := (cid:80) pn
=
S n m , R MPSN ≤ 2 (cid:80) N − 1 j =0 (cid:0) M − 1 j (cid:1) .
Moreover, if m ≤ d n and rank( O n ) =
S n for all n , writing S = max { S 0 , . . .
, S p } , we have the lower bound
R MPSN ≥
N
[Equation]
[Equation] (cid:88) j = (cid:18) M −j (cid:19) . We note that the MPSN lower bound (26) surpasses the SCNN upper bound (22), with order (cid:81) n ( (cid:80) n (cid:48)
mS n (cid:48) )
d n S n and (cid:81) n ( mS n ) d n S n , respectively, when d n ≥ m .
The GNN bound (21) is a special case of the SCNN bound (22) with p = 0 , and has asymptotic order m d 0 S 0 .
It is not difﬁcult to obtain minor case by case improvements of the bounds in Theorem 26 by conducting a more careful analysis of the row independencies in matrix W for speciﬁc values of the input feature dimensions d 0 , . . .
, d p , output feature dimension m , numbers of simplices S 0 , . . .
, S p , and the structure of the matrices U n , M n , O n , n = 0 , . .
. , p .
Proof of Theorem 26.
The proof of the ﬁst upper bound is analogous to Theorem 22.
The difference is now we consider also the faces and cofaces that interact with an n -simplex in the MPSN.
We use the expression (15).
The difference compared with Theorem 22 lies in the number of input features for each n , which here results in
[Equation]
which is the ﬁrst upper bound.
The second upper bound is the trivial upper bound, which is the maximum possible number of regions of a central arrangement of M hyperplanes in R N from
Theorem 20.
The lower bound follows from inserting Proposition 25 into Proposition 24.
The regions for the three network architectures are illustrated in Figure 2 for a complex with
S
= S 1 = 3 and S 2 = 1 , each input dimension 1 and output dimension m = 3 .
It shows that from
GNN, SCNN to MPSN the number of linear regions increases in turn, consistent with the theory.
MPSN with populated higher-features To conclude this section, we consider a situation of interest, where we are given a simplicial complex but only vertex features are available.
To still exploit the structure of the simplicial complex, we can populate the higher features as linear functions of the vertex features.
In this case (18) becomes ψ ( W (cid:48) vec( H in0 )) , where W (cid:48)
=
WC for some matrix C ∈ R ( (cid:80) n d n S n ) × ( d 0 S 0 ) .
We show that this strategy indeed can increase the complexity of the represented functions.
Proposition 27 (MPSN with populated higher-features) .
Consider an arbitrary simplicial complex and an MPSN layer mapping R S
× d 0 → R S 0 × m ; H in0 (cid:55)→ H out0 , whereby higher-dimensional input features are populated as linear functions of the input vertex features.
Consider further the corresponding SCNN layer which computes H out0 =
ψ ( L 0 H in0 W 0 ) .
Then, R MPSN ≥ R SCNN .
Moreover, for certain simplicial complexes and feature dimensions d 0 , . . .
, d p , m , the above inequality is strict.
Proof.
Focusing on the S 0 output features, the relevant matrices are [ W (cid:62)
⊗ M 0 ] for the simplicial network and [ W (cid:62)
⊗ M 0 | W (cid:62) 1 ⊗ O 0 ] C for the MSPN with populated higher-dimensional features.
Both matrices have format mS 0 × dS 0 and rank min { m, d 0 } rank( M 0 ) .
However, subsets of rows have different ranks in both cases.
For illustration, if d = 1 and l is the smallest number of nonzero entries of any row in M 0 , then [ W (cid:62)
M 0 ] has an m row submatrix of rank min { m, l } .
In contrast, an m row submatrix of the augmented matrix will have rank min { m, l + l (cid:48) } , where l (cid:48) is the smallest number of nonzero entries of a row in O 0 .
The regions for the two cases are illustrated in Figure 7.
It shows MPSN (Right) has more regions than GNN/SCNN (Left) and has a higher complexity for populated case.
D R ELATIONSHIP TO C ONVOLUTIONS
Background on Hodge Laplacian
The simplicial convolutions described in this section rely on the Hodge Laplacians of the simplicial complex.
Such a Laplacian operator L p is associated with each dimension p of the complex.
The operator L p has a special structure that depends on the boundary operators of the corresponding dimensions B p and B p +1 .
These matrices are the discrete equivalent of the boundary operators encountered in algebraic topology (see Schaub et al.
(2020) for more details).
They essentially describe things such as the fact that boundary of a (ﬁlled) triangle is [Figure] 1 Regions over 2D slice ofSCNN input space, S=3, d=1, m=3, p=1 pop - -0. [Figure]
d
[Figure]
[Figure]
m=3, p
[Figure]
formed by its edges with some particular choice of orientation (positive or negative).
The Laplacian can be written as a function of these boundary matrices as L p =
B Tp B p + B p +1 B Tp +1 .
We denote the ﬁrst term by L ↓ p and the second by L ↑ p , because they encode (orientation-aware) up and down adjacencies between the p -simplicies of the complex.
Additionally, let us recall the important relation
B p B p +1 =
(i.e. the zero map).
This equation, which is fundamental in homology theory and differential geometry, formally speciﬁes that the boundary of a simplex has no boundary (e.g. the boundary of a ﬁlled triangle has no boundary because it forms a loop and, therefore, it has no endpoints).
We emphasise that the boundary matrices and the Laplacian and, consequently, the convolutions which are based on these Laplacian operators depend on an arbitrary choice of orientation for the simplicial complex.
Therefore, convolutional operators based on this Laplacian using arbitrary nonlinear activations are not generally orientation-equivariant. In other words, changing the orientation of the input complex could result in completely different outputs. We analyse this in more detail in
Appendix E
Simplicial Neural Networks Ebli et al.
(2020) presented Simplicial Neural Networks (SNNs), neural network models extending convolutional layers to attributed simplicial complexes.
The theoretical construction closely resembles the one by Defferrard et al.
(2016), where the standard graph Laplacian is simply replaced by the more general Hodge p -Laplacian
L p , i.e. the generalization of the Laplacian operator to simplices of order p .
The p -th order SNN convolutional layer is
[Equation]
where φ W is a convolutional ﬁlter with learnable parameters W , c
∈ C p ( K ) is a p -cochain on input simplicial complex K (i.e. a real valued function over the set of p -simplices in K ) and ψ accounts for the application of bias and non-linearity.
In particular, the convolutional ﬁlter φ W is parameterized as an R -degree polynomial of the Hodge p -Laplacian
L p .
By imposing a small degree R , it is possible to guarantee spatial ﬁlter localization similarly as in graphs.
This proposed convolutional layer can easily be rewritten in terms of our message passing scheme.
Let us ﬁrst conveniently introduce the following Lemma:
Lemma 28.
The r -th power of the Hodge p -Laplacian, L rp , is equivalent to the sum of the r -th powers of its constituent upper and lower components, that is:
L rp =
( L ↓ p + L ↑ p ) r
=
( L ↓ p ) r
+ ( L ↑ p ) r .
Proof.
We prove the lemma by induction on the power exponent r .
As the base case, we consider exponent r = 1 , for which the equivalence clearly holds as L 1 p =
( L ↓ p + L ↑ p ) 1 =
L ↓ p + L ↑ p =
( L ↓ p )
+ ( L ↑ p ) 1 .
For the induction step, we assume that L r − 1 p
[Equation] − 1 + ( L ↑ p ) r − 1 holds true and prove that
L rp =
( L ↓ p ) r + ( L ↑ p ) r .
L rp is deﬁned as L rp =
L p L r − 1 p
[Equation]
By the induction hypothesis: ( L ↓ p + L ↑ p )( L ↓ p + L ↑ p ) r − 1 = ( L ↓ p + L ↑ p ) (cid:0) ( L ↓ p ) r − 1 + ( L ↑ p ) r − 1 (cid:1) =
L ↓ p ( L ↓ p ) r − 1 + L ↓ p ( L ↑ p ) r − 1 + L ↑ p ( L ↓ p ) r − 1 + L ↑ p ( L ↑ p ) r − 1 =
( L ↓ p ) r + L ↑ p ( L ↓ p ) r − 1 + L ↓ p ( L ↑ p ) r − 1 +( L ↑ p ) r .
The second term L ↑ p ( L ↓ p )
r − 1 is rewritten as B p +1 B Tp +
( B Tp B p ) r − 1 =
B p +1 B Tp +1 B Tp B p
(cid:124) (cid:123)(cid:122) (cid:125) ( r − 1) times = 0 , since B Tp +1 B Tp =
( B p B p +1 ) T , which equals 0 T by deﬁnition.
Similarly, the third term L ↓ p ( L ↑ p ) r −is rewritten as B Tp B p ( B p +1 B Tp +1 ) r − 1 = B Tp B p B p +1 B
Tp
+
(cid:124) (cid:123)(cid:122) (cid:125)
( r − 1) times = 0 , since, again, B p B p +1 = 0 .
We now proceed to prove Theorem 7 that our MPSN can be reduced to SCNN of Ebli et al.
(2020) or
Bunch et al.
We split the proofs in two individuals.
Proof of Theorem 7 in reference to Ebli et al.
We ﬁrst rephrase Equation 22 for a generic layer and multi-dimensional input and output p -simplicial representations
[Equation]
The convolutional operation for p -simplex v can be rewritten as
[Equation]
where h t +1 v denotes the v -th row of matrix H t +1 , ( L rp ) v denotes the v -th row of operator L rp and ( L rp ) v,w its entry at position v, w .
We now leverage on Lemma 28 to rewrite the convolution operation on simplex v as
[Equation]
Considering that matrices L ↓ p and L ↑ p only convey the notions of, respectively, lower and upper simplex adjacency, the equation above is easily interpreted in terms of our message passing scheme by setting
[Equation]
and by letting AGG be the summation over the extended notion of upper and lower R -neighborhoods, that is neighborhoods comprising p -simplices at a distance from v which is at most R ( w is at distance d from v if there exists a sequence of upper- (respectively, lower-) adjacent p -simplices
[ ν 0 , ν 1 , . .
. , ν d ] such that ν 0 = v, ν d = w ).
It is noteworthy that, contrary to our general proposed framework, the two message functions M t +1 ↑ and M t +1 ↓ share the same learnable parameters { W t +1 r } Rr =1 , and that no signal of order lower or higher than p is involved in the computation.
SC-Conv Bunch et al.
(2020) proposed a convolutional operator that can be applied on 2dimensional simplicial complexes.
The construction is based on the canonical normalised Hodge
Laplacians deﬁned by Schaub et al.
(2020); starting from the operators, the authors generalize the
Graph Convolutional Network model proposed in Kipf & Welling (2017) by deﬁning the corresponding adjacency matrices with added self-loops:
[Equation]
We defer readers to Section 2.1 of the original paper for the deﬁnitions of the { ˜ A αi } 2 i =0 and { D } 5 i = matrices in the above equations.
Differently from (Ebli et al., 2020), this scheme models the interactions between signals deﬁned at different dimensions.
It can, nonetheless, be rewritten in terms of our message passing framework.
Proof of Theorem 7 in reference to Bunch et al.
We report here the derivation for message passing on 1 -simplices (edges) as it is the most general.
The derivation on 0 - and 2 -simplices can simply be obtained as a special case of this last.
We ﬁrst denote, for simplicity,
[Equation]
We note that the convolutional operation for a generic 1 -simplex e can be rewritten as
[Equation]
This equation is interpreted in terms of our message passing scheme by setting
[Equation]
and AGG = (cid:80) .
E E QUIVARIANCE AND I NVARIANCE
One would expect MPSNs to be aware of the two symmetries of a simplicial complex: relabeling of the simplicies in the complex and, optionally, changes in the orientation of the complex if orientations are taken into account.
We address these two below.
Let K be simplicial p -complex with boundary matrices
B i with i ∈ { 0 , . . .
, p } and corresponding simplicial attributes X i .
Let P i ∈ R S
i × S
i be some corresponding permutation matrices for the simplicies of dimension i .
Let P
i
X i and P
i − 1 B i P
Ti be the permuted feature matrices and boundary operators, respectively.
Additionally, Let P be an operator acting on simplicial complexes that produces a new complex PK permuted as above.
Deﬁnition 29 (Permutation equivariance and invariance) .
We say that a function f is (simplex) permutation equivariant if f ( PK ) =
P f ( K ) for any P (i.e. for any set of permutation operators { P i } ).
Similarly, we say that a function f is (simplex) permutation invariant if f ( PK ) =
f ( K ) for any P .
Remark 30.
Equivariant functions f must map the simplicial complex to another complex with the same structure, but with possibly different features, similarly to an MPSN layer.
In contrast, an invariant function f is not restricted in its choice of the co-domain, similarly to an MPSN network ending with a readout.
Theorem 31.
An MPSN layer is simplex permutation equivariant and an MPSN network with a ﬁnal readout layer is simplex permutation invariant.
Proof.
It is sufﬁcient to prove that an MPSN layer is permutation equivariant.
Since the ﬁnal readout layer is permutation invariant by deﬁnition, the invariance of the whole model follows directly.
In the proof, we abuse the notation slightly and use P i ( a ) to denote the corresponding permutation function of P i acting on indices.
We focus on a single simplex v of an arbitrary dimension n and the corresponding w = P n ( v ) .
Let h t +1 v be the output feature of simplex v for an MPSN layer taking K as input and ¯ h t +1 w the output features of simplex w for the same MPSN layer taking PK as input.
We will now show they are equal by showing that the multi-set of features being passed to the message, aggregate and update functions are the same for the two simplicies.
The faces of the n -simplicies in K are given by the non-zero elements of B n .
Similarly, the lower adjacencies in PK are given by the non-zero elements of P n − 1 B n P Tn (i.e. the matrix where the rows and columns are permuted according to P n ).
Therefore, we obtain
[Equation]
In particular, this holds for b = v, P n ( b )
= w .
Because the feature matrices for the ( n − 1) -simplices in PK are also permuted with P n − 1 X n − 1 , v and w receive the same message from their faces.
The proof follows similarly for coface adjacencies.
The lower adjacenices of the n -simplicies in K are given by the non-zero entries of B Tn B n .
Similarly, the lower adjacencies in PK are given by the non-zero elements of
[Equation]
That is, the same adjacencies as in K , but with the rows and columns accordingly permuted.
Therefore,
[Equation]
which, in particular, holds for a = v, P n ( a )
= w .
Since the feature matrices for the n -simplices in
PK are also accordingly permuted with P n X n , v and w receive the same message from the lower adjacenct simplicies.
This can be similarly shown for upper adjacencies.
Another symmetry that we would like to preserve is orientation.
For instance, we know that the homology of the complex is invariant to the particular orientation that was chosen.
Therefore, for certain applications where orientations are of interest, we would like to design MPSN layers that are orientation equivariant and MPSN networks that are orientation invariant.
We ﬁrst deﬁne these notions.
Let T
i
=
diag( t i, 1 , . .
, t i,S i ) a set of diagonal matrices where t i,k = ± 1 for any k and i ∈ { 0 , . . .
, p } .
Additionally, we impose the constraint T
=
I , since vertices have no orientation.
Changing the orientation of the complex amounts to obtaining a new set of features T
i X i and boundary matrices T
i − 1 B i T i .
We denote the corresponding change of orientation for the whole complex by T K as above.
Deﬁnition 32 (Orientation equivariance and invariance) .
We say that a function f is orientation equivariant if f ( T K ) =
T f ( K ) for any T (i.e. for any set of orientation operators { T i } ).
Similarly, we say that a function f is orientation invariant if f ( T K ) =
f ( K ) for any T .
Remark 33.
Orientation invariance can be trivially achieved by considering the absolute value of the features and by treating the complex as an unoriented one.
However, it is (in general) desirable to use equivariance at the intermediate layers and make the network invariant with a ﬁnal transformation (readout).
To study such models requires making further assumptions about the structure of the message, update and aggregate functions.
For instance, we can consider the model from (14) used in our linear regions analysis, with a convenient vectorised form
X out
i
[Equation]
where we have split the upper and lower adjacencies in two.
This corresponds to an MPSN with a message function that multiplies the linearly transformed features of the neighbour by the relative orientation ( ± 1 ), sum-based aggregation and an update function that adds the incoming messages to its linearly transformed features and passes the output through ψ .
Proposition 34.
When ψ is an odd activation function, the MPSN layer from Equation (23) is orientation equivariant.
Proof.
We denote with X out i =
f i ( B i , B i +1 , X in i , X in i − 1 , X in i +1 )
the application of one such MPSN layer on i -dimensional chains.
For this MPSN layer to be equivariant, we need to show that f i ( T i − 1 B i T i , T i B
i +1 T
i +1 , T i X in i , T i − 1 X in i − 1 , T {+i +
+}
X in i +1 ) =
T i X out i .
Because T i T i
=
I and T Ti = T i for all i , we can easily rewrite LHS as
[Equation]
in i − 1 W 4 + B i +1 X in i +1 W 5 (cid:1)(cid:17) .
Notice that if ψ and T i commute, then this becomes T i X out i .
We remark that they commute when ψ is an odd function and, in particular, when ψ is the identity.
Remark 35.
A (permutation invariant) aggregation-based readout layer ﬁrst applying an elementwise even function ψ to the elements of its input multiset is orientation invariant since [Equation]
[Equation]
F D ISCUSSION AND R ELATED W ORK F. C OMPUTATIONAL C OMPLEXITY
[-Then, a message passing procedure relying on Theorem 1, which considers only these adjacencies, has a computational complexity Θ (cid:0) (cid:80) pd =
-]
Message passing on clique complexes Generally, ﬁnding all the maximal cliques in a graph (Bron & Kerbosch, 1973; Tomita et al., 2006; Cazals & Karande, 2008) has an optimal worst-case complexity of O (3 n/ 3 ) .
However, ﬁnding all the cliques of a certain maximum dimension in (sparse) real-world graphs can be shown to be signiﬁcantly faster, as shown in the analysis from the main text.
In our experiments, we use a simplex tree data-structure (Boissonnat & Maria, 2014) implemented in the topological data analysis library Gudhi (The GUDHI Project, 2021) to compute the clique complex.
Empirically, in large-scale experiments involving graphs with 10 6 nodes, simplex trees are able to generate the clique complex up to a constant desired dimension in a computational cost that is linear in the number of simplicies in the complex (Boissonnat & Maria, 2014).
Guarantees about the maximum number of such cliques can also be obtained in terms of the maximum degree of the graph, besides its arboricity.
Recently, Chase (2020) has proven the Gan-Loh-Sudakov conjecture (Gan et al., 2015) stating that the maximum number of cliques of size t in a graph of maximum degree δ is q (cid:0) δ +1 t (cid:1) + (cid:0) rt (cid:1) , where n = q ( δ + 1) + r, 0 ≤ r ≤ δ .
This provides further guarantees that the time complexity will be good on real-world graphs, where δ is typically small compared to the size of the graph.
F. O THER L IFTING T RANSFORMATIONS AND C UBICAL C OMPLEXES
We note that other graph lifting transformations could also be used for applying SWL and MPSNs to graph domains.
While clique complexes are the commonest such transformation, many others exist (Ferri et al., 2018) and they could be used to emphasise motifs that are relevant for the task (Milo et al., 2002).
More broadly, the transformation could target a much wider set of complexes such as cubical complexes (Kaczynski et al., 2004) or cell complexes (Hatcher, 2000), for which a message passing procedure has already been proposed (Hajij et al., 2020).
Cell Complexes have a similar hierarchical structure to simplicial complexes, and the approach of Hajij et al.
(2020) is very close to ours.
At the same time, arbitrary cell complexes are too general for most applications.
Here we discuss how our approach could be extended to cubical complexes, a type of cell complex that from a theoretical point of view is similar to simplicial complexes (Kaczynski et al., 2004), and which is used in applications.
We plan to extend our approach to cubical complexes in future work.
Cubical complexes are cell complexes consisting of unions of points, edges, squares, cubes, and higher-dimensional hypercubes (Kaczynski et al., 2004).
While they are less well-known than simplicial complexes, they can nevertheless be used in applications, and are better suited to the study of certain types of data sets, see, e.g. Wagner et al.
Our approach to message passing can be directly implemented also for cubical complexes.
One could similarly deﬁne a WL test for cubical complexes, however, the results of Section 2 do not carry over to cubical complexes, since there exist graphs that cannot be embedded into an n -dimensional cube for any n (Garey & Graham, 1975).
It is therefore not possible to relate the expressive power for graphs in the WL test to cubical complexes.
In Figure 8 we provide an example of two cubical complexes that are not isomorphic, while their underlying graphs are isomorphic.
We note that the clique complexes of the underlying graphs are also isomorphic.
The examples in Figures 5 and 8 raise the question of whether we could design tests that are better suited to take into account the topological information encoded in simplicial and cubical complexes, such as homeomorphism tests, which have already been studied, see e.g. Baik &
Miller (1990).
In particular, such tests should be able to distinguish the graphs in Figure 9, a pair of real world molecular graphs that cannot be distinguished by the SWL test based on clique complexes.
A natural next step in our work is to perform tests on data sets that are more naturally modelled by cubical complexes, such as digital images, for which they can provide computational speed-ups compared to simplicial complexes (Wagner et al., 2011; Kaczynski et al., 2004).
F. P ROVABLY EXPRESSIVE GNN S
In order to overcome the limited expressive power of standard GNN architectures, several works have proposed variants inspired by the higher-order k -WL procedures (see Appendix).
Maron et al.
(2019) introduced a model equivalent in power to 3 -WL, Morris et al.
(2019) proposed k -GNNs, graph neural networks equivalents of set-based k -WL tests.
By performing message passing on all possible k -sets of nodes and across non-local neighborhoods, these models trade locality of computation for expressive power, and thus suffer from high spatial and temporal complexities.
Local k -WL variants are introduced and characterized in their expressive power in a follow-up work (Morris et al., 2020).
These approaches distinguish local and global neighbors and provably powerful neural counterparts are introduced.
Although more efﬁcient than standard higher-order procedures, in contrast to our method, these approaches still account for all possible node k -tuples in a graph, and do not model the relation between signals deﬁned over different dimensions.
An alternative approach to improving
GNN expressivity has been adopted in Bouritsas et al.
(2020), where isomorphism counting of graph substructures is employed as a symmetry breaking mechanism to disambiguate neighbors.
Similarly to ours, this approach retains locality of operations; however, message passing is only performed at the node level, and thus does not account for signals deﬁned over higher-dimensional objects.
G A DDITIONAL E XPERIMENTS AND D ETAILS G. S TRONGLY R EGULAR G RAPHS
In the experiments discussed in Section 3, we consider two graphs to be isomorphic if the Euclidean distance between their representations is below a ﬁxed threshold ε .
We set ε = 0 .
01 and embed graphs in a 16 -dimensional space by running an untrained, 5 -layer, MPSN model on the associated clique complexes.
In particular, each graph is lifted to a ( k − 1) -dimensional simplicial complex, with k the size of the largest clique in the family it belongs to.
Nodes are initialised with a constant, unitary signal, while simplices with the sum of the features of the constituent nodes.
In the employed architecture, in accordance with Theorem 1, messages are only aggregated from faces and upperadjacent simplices (for which we also include the representations of the shared cofaces).
The following message passing operations are employed to compute the t + 1 intermediate representation for p -simplex v :
[Equation]
(26) where (cid:107) indicates concatenation, MLP t is a 2-Layer Perceptron and MLP tU , MLP tM consist of a dense layer followed by a non-linearity.
Parameter (cid:15) is set to zero and is not optimised.
Chain representations are pooled via summation and ﬁnal complex embeddings are obtained by applying an 2-Layer Perceptron to the sum of chain representations at each dimension.
The ELU Clevert et al.
(2016) non-linearity is applied throughout the whole architecture.
The experiments are performed for 10 different random weight-initialisations; in Figure 3 we report mean failure rate along with standard error (vertical error bars).
The “MLP+sum” baseline consists of model which ﬁrst performs a non-linear projection of simplex features, then applies an overall sum readout and ﬁnally applies two additional non-linear projection layers.
ELU nonlinearities are employed for this baseline as well.
The datasets can be found at the webpage http://users.cecs.anu.edu.au/˜bdm/ data/graphs.html .
G. R EAL - WORLD G RAPH C LASSIFICATION
For this set of experiments, we employ a SIN model which applies the following message passing scheme to compute the t + 1 intermediate representation for p -simplex v :
[Equation]
where (cid:107) indicates concatenation, MLP t F ,p and MLP t ↑ ,p are 2-Layers Perceptrons endowed with Batch
Normalization (Ioffe & Szegedy, 2015) (BN) and ReLU activations, MLP tU,p is a dense layer followed by the application of BN and ReLU.
The only exception is represented by Reddit datasets, on which
BN was observed to cause instabilities in the training process and was not applied.
Parameters (cid:15) F and (cid:15) ↑ are set to zero and are not optimised.
As it is possible to notice in Equation (27), upper message m ↑ ,p ( v ) is computed as m ↑ ,p ( v ) =
(cid:80) w ∈N ( v ) h tw , thus explicitly disregarding the representation of shared cofaces h t v ∪ w : this choice showed to yield better performance on these benchmarks.
The overall architecture closely resembles the one adopted in Xu et al.
(2019b): 4 message passing layers and a Jumping Knowledge readout scheme (Xu et al., 2018), which computes ﬁnal p -chain embeddings by applying a non-linear dense layer to the concatenation of the representations obtained at each message passing iteration.
{+Final complex representations are obtained by summing the chain embeddings read-out at dimensions 
+}
(‘nodes’) and 2 (‘triangles’).
One last dense layer is applied to output class predictions.
G. T RAJECTORY C LASSIFICATION
For the trajectory classiﬁcation task we generate the simplicial complex using an approach similar to
Schaub et al.
We generate 1,000 random points in the unit square, we perform a Delaunay triangulation of these points and then remove the triangles (and points) intersecting with two predeﬁned regions of the plane to create the two holes.
To generate the trajectories, we ﬁrst randomly sample a random point from the top-left corner of the complex and an end point from the bottom-right corner of the complex.
We then perform a random walk on the edges of the complex.
With a probability of 0 . 9 , the neighbour closest to the end-point is chosen, and with a probability 0 .
1 , a random neighbour is chosen.
To generate the two classes, we set random points either from the bottom-left corner or the top-right corner as an intermediate checkpoint.
We generate 1,000 train trajectories and 200 test trajectories.
We train all modes for 50 epochs and report the ﬁnal test result.
Model-wise, we use an MPSN network with equivariant layers with identity activation function as in
Equation (23).
The ﬁnal layer is an orientation-invariant readout layer, which ﬁrst passes the features through the absolute value function and then performs a sum aggregation.
For the MPNN baseline we use again a model like Equation (23), but with no upper adjacencies (no triangle awareness) and with no awareness of the relative orientations.
The MPNN is thus essentially performing message passing in the line graph.
[-Final complex representations are obtained by summing the chain embeddings read-out at dimensions 
-]