T OPOLOGICAL G RAPH N EURAL N ETWORKS
A BSTRACT
Graph neural networks (GNNs) are a powerful architecture for tackling graph learning tasks, yet have been shown to be oblivious to eminent substructures such as cycles.
We present TOGL, a novel layer that incorporates global topological information of a graph using persistent homology.
TOGL can be easily integrated into any type of GNN and is strictly more expressive in terms of the Weisfeiler–
Lehman graph isomorphism test.
Augmenting GNNs with TOGL leads to improved predictive performance for graph and node classification tasks, both on synthetic data sets, which can be classified by humans using their topology but not by ordinary GNNs, and on real-world data.
I NTRODUCTION
Graphs are a natural representation of structured data sets in many domains, including bioinformatics, image processing, and social network analysis.
Numerous methods address the two dominant graph learning tasks of graph classification or node classification.
In particular, graph neural networks (GNNs) describe a flexible set of architectures for such tasks and have seen many successful applications over recent years (Wu et al., 2021).
At their core, many GNNs are based on iterative message passing schemes (see Shervashidze and Borgwardt (2009) for an introduction to iterative message passing in graphs and Sanchez-Lengeling et al.
for an introduction to GNNs)
Since these schemes are collating information over the neighbours of every node, GNNs cannot necessarily capture certain topological structures in graphs, such as cycles (Bouritsas et al., 2021).
These structures are highly relevant for applications that require connectivity information, such as the analysis of molecular graphs (Hofer et al., 2020; Swenson et al., 2020).
We address this issue by proposing a Topological Graph Layer (TOGL) that can be easily integrated into any GNN to make it ‘topology-aware.’
Our method is rooted in the emerging field of topological data analysis (TDA), which focuses on describing coarse structures that can be used to study the shape of complex structured and unstructured data sets at multiple scales.
We thus obtain a generic way to augment existing GNNs and increase their expressivity in graph learning tasks.
Figure 1 provides a motivational example that showcases the potential benefits of using topological information: (i) high predictive performance is reached earlier for a smaller number of layers, and (ii) learnable topological representations outperform fixed ones if more complex topological structures are present in a data set.
Our contributions.
We propose TOGL, a novel layer based on TDA concepts that can be integrated into any GNN.
Our layer is differentiable and capable of learning contrasting topological representations of a graph.
We prove that TOGL enhances expressivity of a GNN since it incorporates the ability to work with multi-scale topological information in a graph.
Moreover, we show that
TOGL improves predictive performance of several GNN architectures when topological information is relevant for the respective task.
Numberoflayers/iterations
T e s t acc u r ac
y
( i n % )
GCN
GCN-TOGL
WL
PH (a) C YCLES        25 .50 .75 .100 .
Numberoflayers/iterations
T e s t acc u r ac
y
( i n % )
GCN
GCN-TOGL
WL
PH (b) N ECKLACES
B ACKGROUND : C OMPUTATIONAL T OPOLOGY
We consider undirected graphs of the form G =
( V, E ) with a set of vertices V and a set of edges E ⊆
V × V .
The basic topological features of such a graph G are the number of connected components β 0 and the number of cycles β 1 .
These counts are also known as the 0 -dimensional and 1 -dimensional Betti numbers , respectively; they are invariant under graph isomorphism (Hatcher, 2002, pp.
103–133) and can be computed efficiently.
The expressivity of Betti numbers can be increased using a graph filtration , i.e. a sequence of nested subgraphs of G such that ∅ = G (0) ⊆
G (1) ⊆
G (2) ⊆ · · · ⊆ G ( n − 1) ⊆
G ( n ) =
G .
A filtration makes it possible to obtain more insights into the graph by ‘monitoring’ topological features of each G ( i ) and calculating their topological relevance, also referred to as their persistence .
If a topological feature appears for the first time in G ( i ) and disappears in G ( j ) , we assign this feature a persistence of j − i .
Equivalently, we can represent the feature as a tuple ( i, j ) , which we collect in a persistence diagram D .
If a feature never disappears, we represent it by a tuple ( i, ∞ ) ; such features are the ones that are counted for the respective Betti numbers.
This process was formalised and extended to a wider class of structured data sets, namely simplicial complexes, and is known under the name of persistent homology .
One of its core concepts is the use of a filtration function f : V → R d , with d = 1 typically, to accentuate certain structural features of a graph.
This replaces the aforementioned tuples of the form ( i, j ) by tuples based on the values of f , i.e. ( f i , f j ) .
Persistent homology has shown excellent promise in different areas of machine learning research (see Hensel et al.
(2021) for a recent survey and Appendix A for a more technical description of persistent homology), with existing work stressing that choosing or learning an appropriate filtration function f is crucial for high predictive performance (Hofer et al., 2020;
Zhao and Wang, 2019).
Notation.
We denote the calculation of persistence diagrams of a graph G under some filtration f by ph ( G, f ) .
This will result in two persistence diagrams D (0) , D (1) , containing information about topological features in dimension 0 (connected components) and dimension 1 (cycles).
The cardinality of D (0) is equal to the number of nodes n in the graphs and each tuple in the 0 -dimensional diagram is associated with the vertex that created it.
The cardinality of D (1) is the number of cycles; we pair each tuple in D (1) with the edge that created it.
Unpaired edges—edges that are not used to create a cycle—are assigned a ‘dummy’ tuple value, such as (0 , 0) .
All other edges will be paired with the maximum value of the filtration, following previous work by Hofer et al.
R ELATED W ORK
Graph representation learning has received a large amount of attention by the machine learning community.
Graph kernel methods address graph classification via (implicit or explicit) embeddings in Reproducing Kernel Hilbert Spaces (Borgwardt et al., 2020; Kriege et al., 2020; Nikolentzos et al., 2019).
While powerful and expressive, they cannot capture partial similarities between neighbour- hoods.
This can be achieved by graph neural networks , which typically employ message passing on graphs to learn hidden representations of graph structures (Kipf and Welling, 2017; Wu et al., 2021).
Recent work in this domain is abundant and includes attempts to utilise additional substructures (Bouritsas et al., 2021) as well as defining higher-order message passing schemes (Morris et al., 2019) or algorithms that generalise message passing to more complex domains (Bodnar et al., 2021).
Our approach falls into the realm of topological data analysis (Edelsbrunner and Harer, 2010) and employs persistent homology , a technique for calculating topological features—such as connected components and cycles—of structured data sets.
These features are known to be highly characteristic, leading to successful topology-driven graph machine learning approaches (Hofer et al., 2017; 2020;
Rieck et al., 2019; Zhao and Wang, 2019).
At their core is the notion of a filtration , i.e. a sequence of nested subgraphs (or simplicial complexes in a higher-dimensional setting).
Choosing the right filtration is known to be crucial for obtaining good performance (Zhao and Wang, 2019).
This used to be a daunting task because persistent homology calculations are inherently discrete.
Recent advances in proving differentiability enable proper end-to-end training of persistent homology (Carri`ere et al., 2021), thus opening the door for hybrid methods of increased expressivity by integrating the somewhat complementary view of topological features.
Our method TOGL builds on the theoretical framework by Hofer et al.
(2020), who (i) demonstrated that the output of a GNN can be used to ‘learn’ one task-specific filtration function, and (ii) described the conditions under which a filtration function f is differentiable.
This work culminated in GFL, a topological readout function that exhibited improved predictive performance for graph classification tasks.
We substantially extend the utility of topological features by making existing GNNs ‘topology-aware’ through the development of a generic layer that makes topological information available to all downstream GNN layers: TOGL can be integrated into any GNN architecture, enabling the creation of hybrid models whose expressivity is provably more powerful than that of a GNN alone.
Moreover, while GFL only uses the output of a GNN to drive the calculation of topological features by means of a single filtration (thus limiting the applicability of the approach, as the topological features cannot inform the remainder of a network), TOGL learns multiple filtrations of a graph in an end-to-end manner.
More precisely,
TOGL includes topological information in the hidden representations of nodes, enabling networks to change the importance of the topological signal.
Closest to the scope of TOGL is Zhao et al.
(2020), who enhanced GNNs using topological information for node classification.
In their framework, however, topology is only used to provide additional scalar-valued weights for the message passing scheme, and topological features are only calculated over small neighbourhoods, making use of a static vectorisation technique of persistence diagrams.
Similarly, Wong and Vong (2021) use static, i.e. non-learnable, topological features for 3D shape segmentation.
By contrast, TOGL, being end-to-end differentiable, is more general and permits the calculation of topological features at all scales—including graph-level features—as well as an integration into arbitrary GNN architectures.
TOGL: A T OPOLOGICAL G RAPH L AYER
TOGL is a new type of graph neural network layer that is capable of utilising multi-scale topological information of input graphs.
In this section, we give a brief overview of the components of this layer before discussing algorithmic details, theoretical expressivity, computational complexity, and limitations.
Figure 2 presents an overview of our method (we show only a single graph being encoded, but in practice, the layer operates on batches of graphs).
The layer takes as input a graph G =
( V, E ) equipped with a set of n vertices V and a set of edges E , along with a set of d -dimensional node attribute vectors x ( v ) ∈ R d for v ∈ V .
These node attributes can either be node features of a data set or hidden representations learnt by some GNN.
We employ a family of k vertex filtration functions of the form f i : R d → R for i = 1 , . . .
, k .
Each filtration function f
i can focus on different properties of the graph.
The image of f i is finite and results in a set of node values a (1) i < · · · < a ( n ) i such that the graph G is filtered according to ∅ =
G (0) i ⊆
G (1) i ⊆ · · · ⊆ G ( n ) i = G , where G ( j ) i = (cid:16) V ( j ) i , E ( j ) i (cid:17) , with V ( j ) i := (cid:110) v ∈ V | f i (cid:0) x ( v ) (cid:1) ≤ a ( j ) i (cid:111) , and E ( j ) i := (cid:110) v, w ∈ E | max (cid:8) f
i (cid:0) x ( v ) (cid:1) , f i (cid:0) x ( w ) (cid:1)(cid:9) ≤ a ( j ) i (cid:111) .
Given this filtration, we calculate a set of persistence diagrams, i.e. ph ( G, f i ) =
(cid:110) D (0) i , . . .
, D ( l ) i (cid:111) .
We fix l = 1 (i.e. we are capturing connected components and cycles) to simplify our current implementation, but our layer can be x ( v ) ∈
R d (a)nodeattributes x ( v ) x ( v ) a ( v ) 1 ... a ( v ) k
(b)nodemap Φ 22 1 = a ( v ) k 1 ...
31 2 = a ( v ) 1(c) k viewsofgraph G (d)filtrations f i ...
Ψ (e)Persistencediagrams Ψ[ v ] + x ( v ) ˜
x
( v ) (f)aggregation ˜ x
( v ) ∈ R d (g)output ˜ x
( v ) extended to arbitrary values of l (see Appendix C for a discussion). In order to benefit from representations that are trainable end-to-end, we use an embedding function Ψ ( l ) : (cid:110) D ( l ) 1 , . . .
, D ( l ) k (cid:111)
→ R n ′ ×
d for embedding persistence diagrams into a high-dimensional space that will be used to obtain the vertex representations, where n ′ is the number of vertices n if l = 0 and the number of edges if l = 1 .
This step is crucial as it enables us to use the resulting topological features as node features, making
TOGL a layer that can be integrated into arbitrary GNNs.
We later explain the precise mapping of Ψ ( l ) from a set of diagrams to the elements of a graph.
Details on filtration computation and output generation.
We compute our family of k vertex-based filtrations using Φ: R d → R k , an MLP with a single hidden layer, such that f i :=
π i ◦ Φ , i.e. the projection of Φ to the i th dimension.
We apply Φ to the hidden representations x ( v ) of all vertices in the graph.
Moreover, we treat the resulting persistence diagrams in dimension 0 and 1 differently.
For dimension 0 , we have a bijective mapping of tuples in the persistence diagram to the vertices of the graph, which was previously exploited in topological representation learning (Moor et al., 2020).
Therefore, we aggregate Ψ (0) with the original node attribute vector x ( v ) of the graph in a residual fashion, i.e. ˜ x ( v ) =
x ( v ) + Ψ (0) (cid:16) D (0)1 , . . .
, D (0) k (cid:17) (cid:2) v (cid:3) , where Ψ (0) [ v ] denotes taking v th row of Ψ (0) (i.e the topological embedding of vertex v ).
The output of our layer for dimensiontherefore results in a new representation ˜ x
( v ) ∈ R d for each vertex v , making it compatible with any subsequent (GNN) layers.
By contrast, Ψ (1) is pooled into a graph-level representation, to be used in the final classification layer of a GNN.
This is necessary because there is no bijective mapping to the vertices, but rather to edges.
For stability reasons (Bendich et al., 2020), we consider it more useful to have this information available only on the graph level.
For further details on the computational aspects, please refer to Section A.4.
Complexity and limitations.
Persistent homology can be calculated efficiently for dimensionsand 1 , having a worst-case complexity of O ( mα ( m )) for a graph with m sorted edges, where α ( · ) is the extremely slow-growing inverse Ackermann function, which can be treated as constant for all intents and purposes.
The calculation of ph is therefore dominated by the complexity of sorting all edges, i.e. O ( m log m ) , making our approach efficient and scalable.
Higher-dimensional persistent homology calculations unfortunately do not scale well, having a worst-case complexity of O (cid:0) m d (cid:1) for calculating d -dimensional topological features, which is why we restrict ourselves to l = 1 here.
Our approach is therefore limited to connected components and cycles.
Plus, our filtrations are incapable of assessing topological feature interactions; this would require learning multifiltrations (Carlsson et al., 2009), which do not afford a concise, efficient representation as the scalar-valued filtrations discussed in this paper.
We therefore leave their treatment to future work.
4. C HOOSING AN E MBEDDING F UNCTION Ψ
The embedding function Ψ influences the resulting representation of the persistence diagrams calculated by our approach.
It is therefore crucial to pick a class of functions that are sufficiently powerful to result in expressive representations of a persistence diagram D .
We consider multiple types of embedding functions Ψ , namely (i) a novel approach based on DeepSets (Zaheer et al., 2017), (ii) the rational hat function introduced by Hofer et al.
(2019), as well as (iii) the triangle point transformation, (iv) the Gaussian point transformation, and (v) the line point transformation, with the last three transformations being introduced in Carri`ere et al.
Except for the deep sets approach, all of these transformations are local in that they apply to a single point in a persistence diagram without taking the other points into account.
These functions can therefore be decomposed as Ψ ( j ) (cid:16) D ( j ) 1 , . . .
, D ( j ) k (cid:17) (cid:2) v (cid:3)
= (cid:101) Ψ
( j ) (cid:
) D ( j ) 1 [ v ] , . . .
, D ( j ) k
[ v ] (cid:17) for an index v .
By contrast, our novel deep sets approach uses all tuples in the persistence diagrams to compute embeddings.
In practice, we did not find a significant advantage of using any of the functions defined above; we thus treat them as a hyperparameter and refer to Appendix I for a detailed analysis.
4. D IFFERENTIABILITY & E XPRESSIVE P OWER
The right choice of Ψ will lead to a differentiable downstream representation.
The map ph ( · ) was shown to be differentiable (Gameiro et al., 2016;
Hofer et al., 2020; Moor et al., 2020; Poulenard et al., 2018), provided the filtration satisfies injectivity at the vertices.
We have the following theorem, whose proof is due to Hofer et al.
(2020, Lemma 1).
Theorem 1.
Let f θ be a vertex filtration function f θ : V → R with continuous parameters θ , and let Ψ be a differentiable embedding function of unspecified dimensions.
If the vertex function values of f θ are distinct for a specific set of parameters θ ′ , i.e. f θ ( v ) ̸
= f θ ( w ) for v ̸ = w , then the map θ (cid:55)→ Ψ ( ph ( G, f θ )) is differentiable at θ ′ .
This theorem is the basis for TOGL, as it states that the filtrations, and thus the resulting ‘views’ on a graph G , can be trained end-to-end.
While Hofer et al.
(2020) describe this for a single filtration, their proof can be directly extended to multiple filtrations as used in our approach.
The expressive power of graph neural networks is well-studied (Chen et al., 2020b; Xu et al., 2019) and typically assessed via the iterative Weisfeiler–Lehman label refinement scheme, denoted as
WL[ 1 ].
Given a graph with an initial set of vertex labels, WL[ 1 ] collects the labels of neighbouring vertices for each vertex in a multiset and ‘hashes’ them into a new label, using a perfect hashing scheme so that vertices/neighbourhoods with the same labels are hashed to the same value.
This procedure is repeated and stops either when a maximum number of iterations has been reached or no more label updates happen.
The result of each iteration h of the algorithm for a graph G is a feature vector ϕ ( h ) G that contains individual label counts.
Originally conceived as a test for graph isomorphism (Weisfeiler and Lehman, 1968), WL[ 1 ] has been successfully used for graph classification (Shervashidze and Borgwardt, 2009).
The test runs in polynomial time, but is known to fail to distinguish between certain graphs, i.e. there are non-isomorphic graphs G and G ′ that obtain the same labelling by WL[ 1 ] F ¨ urer (2017).
Surprisingly, Xu et al.
(2019) showed that standard graph neural networks based on message passing are no more powerful than WL[ 1 ].
Higher-order refinement schemes, which pass information over tuples of nodes, for instance, can be defined (Maron et al., 2019; Morris et al., 2019).
Some of these variants are strictly more powerful (they can distinguish between more classes of graphs) than WL[ 1 ] but also computationally more expensive.
To prove the expressivity of our method, we will show that (i) TOGL can distinguish all the graphs
WL[ 1 ] can distinguish, and (ii) that there are graphs that WL[ 1 ] cannot distinguish but TOGL can.
The higher expressivity of TOGL does not necessarily imply that our approach will perform generally better.
In fact, WL[ 1 ] and, by extension, GNNs, are capable of identifying almost all non-isomorphic graphs (Babai et al., 1980).
However, the difference in expressivity implies that TOGL can capture features that cannot be captured by GNNs, which can improve predictive performance if those features cannot be obtained otherwise.
Since persistent homology is an isomorphism invariant, we first show that we distinguish the same graphs that WL[ 1 ] distinguishes.
We do this by showing the existence of an injective filtration function f 1 , thus ensuring differentiability according to Theorem 1.
Theorem 2.
Persistent homology is at least as expressive as WL[ 1 ], i.e. if the WL[ 1 ] label sequences for two graphs G and G ′ diverge, there exists an injective filtration f such that the corresponding 0 -dimensional persistence diagrams D 0 and D ′ 0 are not equal.
Proof sketch.
We first assume the existence of a sequence of WL[ 1 ] labels and show how to construct a filtration function f from this.
While f will result in persistence diagrams that are different, thus serving to distinguish G and G ′ , it does not necessarily satisfy injectivity.
We therefore show that there is an injective function ˜ f
that is arbitrarily close to f and whose corresponding persistence diagrams (cid:102) D 0 , (cid:102)
D ′ 0 do not coincide.
Please refer to Appendix B for the detailed version of the proof.
The preceding theorem proves the existence of such a filtration function.
Due to the capability of GNNs to approximate the Weisfeiler–Lehman test (Xu et al., 2019) and the link between graph isomorphism testing and universal function approximation capabilities (Chen et al., 2019, Theorem 4), we can deduce that they are also able to approximate f and ˜ f , respectively.
Yet, this does not mean that we always end up learning f or ˜ f .
This result merely demonstrates that our layer can theoretically perform at least as well as WL[ 1 ] when it comes to distinguishing non-isomorphic graphs; this does not generally translate into better predictive performance, though.
In practice, TOGL may learn other filtration functions; injective filtrations based on WL[ 1 ] are not necessarily optimal for a specific task (we depict some of the learnt filtrations in Appendix G).
To prove that our layer is more expressive than a GCN, we show that there are pairs of graphs
G, G ′ that cannot be distinguished by WL[ 1 ] but that can be distinguished by ph ( · ) and by TOGL, respectively: let G be a graph consisting of the disjoint union of two triangles, i.e.
, and let G ′ be a graph consisting of a hexagon,
i.e.
WL[ 1 ] will be unable to distinguish these two graphs because all multisets in every iteration will be the same.
Persistent homology, by contrast, can distinguish G from G ′ using their Betti numbers.
We have β 0 ( G ) =
β 1 ( G ) = 2 , because G consists of two connected components and two cycles, whereas β 0 ( G ′ )
=
β 1 ( G ′ )
= 1 as G ′ only consists of one connected component and one cycle.
The characteristics captured by persistent homology are therefore different from the ones captured by WL[ 1 ].
Together with Theorem 2, this example implies that persistent homology is strictly more powerful than WL[ 1 ] (see Appendix C for an extended expressivity analysis using higher-order topological features).
The expressive power of TOGL hinges on the expressive power of the filtration—making it crucial that we can learn it.
E XPERIMENTS
We showcase the empirical performance of TOGL on a set of synthetic and real-world data sets, with a primary focus on assessing in which scenarios topology can enhance and improve learning on graph.
Next to demonstrating improved predictive performance for synthetic and structurebased data sets (Section 5.2 and Section 5.3), we also compare TOGL with existing topology-based algorithms (Section 5.5).
5. E XPERIMENTAL S ETUP
Following the setup of Dwivedi et al.
, we ran all experiments according to a consistent training setup and a limited parameter budget to encourage comparability between architectures.
For further details, please refer to Appendix D. In all tables and graphs, we report the mean test accuracy along with the standard deviation computed over the different folds.
All experiments were tracked (Biewald, 2020); experimental logs, reports, and code will be made publicly available.
Baselines and comparison partners.
We compare our method to several GNN architectures from the literature, namely (i) Graph Convolutional Networks (Kipf and Welling, 2017, GCN), (ii) Graph
Attention Networks (Veliˇckovi´c et al., 2018, GAT), (iii) Gated-GCN (Bresson and Laurent, 2017), (iv) Graph Isomorphism Networks (Xu et al., 2019, GIN), (v) the Weisfeiler–Lehman kernel (Shervashidze and Borgwardt, 2009, WL), and (vi) WL-OA (Kriege et al., 2016).
The performance of these methods has been assessed in benchmarking papers (Borgwardt et al., 2020; Dwivedi et al., 2020; Morris et al., 2020), whose experimental conditions are comparable.
We use the same folds and hyperparameters as in the corresponding benchmarking papers to ensure a fair comparison.
TOGL setup.
We add our layer to existing GNN architectures, replacing the second layer by TOGL, respectively.
For instance, GCN-4 refers to a GCN with four layers, while GCN-3-TOGL-1 refers to method that has been made ‘topology-aware’ using TOGL.
This setup ensures that both the original and the modified architecture have approximately the same number of parameters.
We treat the choice of the embedding function Ψ as a hyperparameter in our training for all subsequent experiments.
Appendix I provides a comprehensive assessment of the difference between the
DeepSets approach (which is capable of capturing interactions between tuples in a persistence diagram) and decomposed embedding functions, which do not account for interactions.
5. P ERFORMANCE ON S YNTHETIC D ATA S ETS
As an illustrative example depicted in Figure 1, we use two synthetic balanced 2-class data sets of 1000 graphs each.
In the C YCLES data set (Figure 1a, right), we generate either one large cycle (class 0 ) or multiple small ones (class 1 ).
These graphs can be easily distinguished by any topological approach because they differ in the number of connected components and the number of cycles.
For the N ECKLACES data set (Figure 1b, right), we generate ‘chains’ of vertices with either two individual cycles, or a ‘merged’ one.
Both classes have the same number of cycles, but the incorporation of additional connectivity information along their neighbourhoods makes them distinguishable for TOGL, since appropriate filtrations for the data can be learnt.
All synthetic data sets use node features consisting of 3-dimensional vectors sampled from a Normal distribution.
As
PH and WL would consider all instances of graphs being distinct and thus remove any potential signal, we used the node degrees as features instead as commonly done (Borgwardt et al., 2020).
For this illustrative example, we integrate TOGL with a GCN.
We find that TOGL performs well even without any GCN layers—thus providing another empirical example of improved expressivity.
Moreover, we observe that standard GCNs require at least four layers (for C YCLES ) or more (for
N ECKLACES ) to approach the performance of TOGL.
WL[ 1 ], by contrast, fails to classify C YCLES and still exhibits a performance gap to the GCN for N ECKLACES , thus showcasing the benefits of having access to a learnable node representation.
It also underscores the observations in Section 4.2: the higher expressive power of WL[ 1 ] as compared to a standard GCN does not necessarily translate to higher predictive performance.
Appendix F provides an extended analysis with more configurations; we find that (i) both cycles and connected components are crucial for reaching high predictive performance, and (ii) the static variant is performing slightly better than the standard GCN, but significantly worse than TOGL, due to its very limited access to topological information.
A simple static filtration (based on node degrees), which we denote by PH, only works for the C YCLES data set (this is a consequence of the simpler structure of that data set, which can distinguished by
Betti number information already), whereas the more complex structure of the N ECKLACES data necessitates learning a task-based filtration.
5. S TRUCTURE - BASED G RAPH AND N ODE C LASSIFICATION P ERFORMANCE
A recent survey (Borgwardt et al., 2020) showed that the node features of benchmark graphs already carry substantial information, thus suppressing the signal carried by the graph structure itself to some extent.
This motivated us to prepare a set of experiments in which we classify graphs solely based on graph structure .
We achieve this by replacing all node labels and node features by random ones, thus leaving only structural/topological information for the classification task.
In the case of the P ATTERN dataset we use the original node features which are random by construction (Dwivedi et al., 2020).
Table 1 depicts the results for graph and node classification tasks on such graphs; we observe a clear advantage of TOGL over its comparison partners: making an existing GNN topology-aware via
TOGL improves predictive performance in virtually all instances.
In some cases, for instance when comparing GCN-4 to GCN-3-TOGL-1 on MNIST, the gains are substantial with an increase of more than 8%.
This also transfers to other model architectures, where in most cases TOGL improves the performance across datasets.
Solely the GIN model on P ROTEINS and the GAT model on P ATTERN decrease in performance when incorporating topological features.
The deterioration of TOGL on
P ATTERNS is especially significant.
Nevertheless, the inferior performance is in line with the low performance of GAT in general compared to the other methods we considered.
In this context the lower performance of TOGL is not surprising as it relies on the backbone model for the construction of a filtration.
This demonstrates the utility of TOGL in making additional structural information available to improve classification performance.
[Table]
are cited from the literature, i.e. Dwivedi et al.
, Morris et al.
(2020) for
IMDB-B and REDDIT-B, and Borgwardt et al.
(2020) for WL/WL-OA results.
Graph classification results are shown on the left; node classification results are shown on the right.
Following Table 1, we take existing architectures and replace their second layer by TOGL; we use italics to denote the winner of each comparison. A bold value indicates the overall winner of a column, i.e. a data set.
[Table]
[Figure]
[Table]
[Figure]
Varying the number of layers in this experiment underscores the benefits of including TOGL in a standard GCN architecture.
Specifically, an architecture that incorporates TOGL reaches high predictive performance earlier, i.e. with fewer layers, than other methods, thus reducing the risk of oversmoothing (Chen et al., 2020a).
Figure 3 depicts this for
MNIST; we observe similar effects on the other data sets.
Appendix I presents a detailed performance comparison ,and extended analysis, and a discussion of the effects of different choices for an embedding function Ψ .
5. P ERFORMANCE ON B ENCHMARK D ATA S ETS
Having ascertained the utility of TOGL for classifying data sets with topological information, we now analyse the effects of TOGL in standard graph and node classification tasks.
Table 2 depicts the results on well-known benchmark data sets for graph and node classification.
We see that
TOGL performs better than its comparison partners (i.e. GCN-4, GIN-4, GAT-4) on most of the data sets, showcasing the benefits of substituting a layer of GNN with TOGL.
Concerning ENZYMES performance, we experienced a severe degree of overfitting during training.
This was exacerbated
[Table]
by the fact that ENZYMES is the smallest of the compared data sets and we eschewed the tuning of regularisation hyperparameters such as ‘dropout’ for the sake of being comparable with the benchmark results (Dwivedi et al., 2020).
With the GAT-based GNN
[Table]
we generally observe an high degree of variance, as already reported in previous studies (Dwivedi et al., 2020).
In particular [Table] we experienced issues in training it on the REDDIT-B dataset.
The addition of TOGL seems to address this issue [Table] which we consider to underline the overall potential of topological features (at least for data sets with distinct topological information).
5. C OMPARISON TO O THER T OPOLOGY -B ASED A LGORITHMS
In light of TOGL containing existing embedding functions Ψ (Carri`ere et al., 2020)
[Table]
we compare its performance to other topology-based algorithms that have been used for graph classification.
Tablesummarises the performance (for comparison purposes, we also show the results for a ‘static’ variant of our layer; see Appendix D for more details).
In order to permit a fair comparison, we integrate
TOGL into a simpler GNN architecture, consisting of a single GCN message passing layer.
We observe that our filtration learning approach outperforms the fixed filtrations used by Carri ` ere et al.
(2020), highlighting the utility of a layer specifically designed to be integrated into GNNs.
TOGL also fares well in comparison to the readout function by Hofer et al.
While large standard deviations (due to the small size of the data sets) preclude an assessment of significant differences, we hypothesise that the benefits of having access to multiple filtrations will be more pronounced for large data sets containing more pronounced topological features, such as molecular graphs.
Notice that in contrast to Hofer et al.
(2020), our layer can be included at different stages of the overal GNN architecture.
We investigate the impact of different positions in Appendix H and show that different positions can lead significantly different performance.
These results further motivate the importance of a flexible topological layer as opposed to readout functions or static filtrations.
C ONCLUSION
We presented TOGL, a generically-applicable layer that incorporates topological information into any
GNN architecture.
We proved that TOGL, due to its filtration functions (i.e. input functions) being learnable, is more expressive than WL[ 1 ], the Weisfeiler–Lehman test for graph isomorphism, and therefore also increases expressivity of GNNs.
On data sets with pronounced topological structures, we found that our method helps GNNs obtain substantial gains in predictive performance.
We also saw that the choice of function for embedding topological descriptors is crucial, with embedding functions that can handle interactions between individual topological features typically performing better than those that cannot.
On benchmark data sets, we observed that our topology-aware approach can help improve predictive performance while overall exhibiting favourable performance in comparison to the literature.
We also observed that topological information may sometimes lead to overfitting issues on smaller data sets, and leave the investigation of additional regularisation strategies for our method for future work.
Furthermore, we hypothesise that the use of different filtration types (Milosavljevi´c et al., 2011), together with advanced persistent homology algorithms, such as extended persistence (CohenSteiner et al., 2009), will prove beneficial for predictive performance.
R EPRODUCIBILITY S TATEMENT
We have provided the code for our experiments along with the seeds used for training.
All experiments were run on single GPUs, which avoids additional sources of randomness.
Details are provided in Section 5.1.
The parameters and performance metrics during training were tracked and will be made publicly available (Biewald, 2020).
Our code is released under a BSD-3-Clause License and can be accessed under https://github.com/BorgwardtLab/TOGL .
A CKNOWLEDGEMENTS
The authors would like to thank “Weights and Biases, Inc.” for providing us with a free academic team account.
This project was supported by the grant #2017-110 of the Strategic Focal Area “Personalized
Health and Related Technologies (PHRT)” of the ETH Domain for the SPHN/PHRT Driver Project “Personalized Swiss Sepsis Study” and the Alfried Krupp Prize for Young University Teachers of the
Alfried Krupp von
Bohlen und Halbach-Stiftung (K.B.) Edward De Brouwer gratefully acknowledges support by an FWO-SB grant and support from NVIDIA for GPUs.
Yves Moreau is funded by (1)
Research Council KU Leuven: C14/18/092 SymBioSys3; CELSA-HIDUCTION,
(2) Innovative
Medicines Initiative: MELLODY, (3) Flemish Government (ELIXIR Belgium, IWT, FWO 06260) and (4) Impulsfonds AI: VR 2019
DOC.0318/1QUATER Kenniscentrum Data en Maatschappij.
Some computational resources and services used in this work were provided by the VSC (Flemish
Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish
Government – department EWI.
A T OPOLOGICAL D ATA A NALYSIS
We provide a more formal introduction to persistent homology, the technique on which TOGL is fundamentally based.
Persistent homology arose as one of the flagship approaches in the field of computational topology, which aims to make methods from this highly abstract branch of mathematics available for data analysis purposes.
To gain a better understanding, we will briefly take a panoramic tour through algebraic topology, starting from simplicial homology , an algebraic technique for ‘calculating’ the connectivity of topological spaces, represented in the form of simplicial complexes, i.e. generalised graphs.
Simplicial homology is said to assess the connectivity of a topological space by ‘counting its high-dimensional holes’.
We will see how to make this description more precise.
A. S IMPLICIAL H OMOLOGY
Simplicial complexes are the central concept in algebraic topology.
A simplicial complex K consists of a set of simplices of certain dimensions, such as vertices (dimension 0 ), edges (dimension 1 ), and triangles (dimension 2 ).
Each simplex σ ∈ K has a set of faces, and each face τ has to satisfy τ ∈ K .
Moreover, if σ ∩ σ ′ ̸ = ∅ for σ, σ ′ ∈ K , then σ ∩ σ ′ ∈ K .
Thus, K is ‘closed under calculating the faces of a simplex’.
A graph can be seen as a low-dimensional simplicial complex that only contains 0 -simplices (vertices) and 1 -simplices (edges), so everything we say applies, mutatis mutandis , also to graphs.
Chain groups.
For a simplicial complex K , we denote by C d (K) the vector space generated over Z 2 (the field with two elements).
The elements of C d (K) are the d -simplices in K , or rather their formal sums with coefficients in Z 2 .
For example, σ + τ is an element of the chain group, also called a simplicial chain .
Addition is well-defined and easy to implement as an algorithm since a simplex can only be present or absent over Z 2 coefficients.
The use of chain groups lies in providing the underlying vector space to formalise boundary calculations over a simplicial complex.
The boundary calculations, in turn, are necessary to quantify the connectivity!
Boundary homomorphism.
Given a d -simplex σ = ( v 0 , . .
. , v d ) ∈ K , we can formalise its face or ‘boundary’ calculation by defining the boundary operator ∂ d : C d (K) → C d − 1 (K) as a sum of the form
[Equation]
i.e. we leave out every vertex v i of the simplex once.
Since only sums are involved, this operator is seen to be a homomorphism between the chain groups; the calculation extends to C d (K) by linearity.
The boundary homomorphism gives us a way to precisely define what we understand by connectivity.
To this end, note that its kernel and image are well-defined.
The kernel ker ∂
d contains all d -dimensional simplicial chains that do not have a boundary.
We can make this more precise by using a construction from group theory.
Homology groups.
The last ingredient for the connectivity analysis involves calculating a special group, the homology group.
The d th homology group H d (K) of K is defined as the quotient group
H d (K) := ker ∂
d / im ∂
d +1 .
The quotient operation can be thought of as calculating a special subset— the kernel of the boundary operator—and then removing another subset, namely the image of the boundary operator with an increased dimension.
This behoves a short explanation.
The main reason behind this operation is that just the kernel calculation is insufficient to properly count a hole.
For example, if we take the three edges of any triangle, their boundary will always be empty, i.e. they are a part of ker ∂ 1 .
However, if the interior of the triangle is also part of the simplicial complex—in other words, if we have the corresponding 2 -simplex as well—we should not count the edges as a hole.
This is why we need to remove all elements in the image of ∂ 2 .
Coincidentally, this also explains why cycles in a graph never ‘vanish’—there are simply no 2 -simplices available since the graph is only a 1 -dimensional simplicial complex.
Betti numbers.
To fully close the loop, as it were, it turns out that we can calculate Betti numbers from homology groups.
Specifically, the rank of the d th homology group—in the group-theoretical sense—is precisely the d th Betti number β d , i.e. β d (K) :
=
rank H d (K) .
The sequence of Betti numbers β 0 , . . .
, β d of a d -dimensional simplicial complex is commonly used as a complexity measure, and they can be used to discriminate manifolds.
For example, a 2 -sphere has Betti numbers (1 , 0 , 1) , while a 2 -torus has Betti numbers (1 , 2 , 1) .
As we outlined in the main text, Betti numbers are of limited use when dealing with complex graphs, however, because they are very coarse counts of features.
It was this limited expressivity that prompted the development of persistent homology.
A. P ERSISTENT H OMOLOGY
Persistent homology is an extension of simplicial homology, which employs filtrations to imbue a simplicial complex K with scale information.
Let us assume the existence of a function f : K → R , which only attains a finite number of function values f (0) ≤ f (1) ≤ · · · ≤ . . .
f ( m − 1) ≤ f ( m ) .
We may now, as in the main text, sort K according to f , leading again to a nested sequence of simplicial complexes
[Equation]
in which K ( i ) :=
[Equation]
σ ∈ K | f ( σ ) ≤ f ( i ) (cid:9) , i.e. each subset contains only those simplices whose function value is less than or equal to the threshold.
In contrast to simplicial homology, the filtration holds potentially more information because it can track changes !
For example, a topological feature might be created (a new connected component might arise) or destroyed (two connected components might merge into one), as we go from some K ( i ) to K ( i +1) .
At its core, persistent homology is ‘just’ a way of tracking topological features, representing each one by a creation and destruction value ( f ( i ) , f ( j ) )
∈ R 2 , with i ≤ j .
In case a topological feature is still present in K ( m )
=
K , such as a cycle in a graph, it can also be assigned a tuple of the form ( f ( i ) , ∞ ) .
Such tuples constitute essential features of a simplicial complex and are usually assigned a large destruction value or treated separately in an algorithm (Hofer et al., 2017).
While it is also possible to obtain only tuples with finite persistence values, a process known as extended persistence (Cohen-Steiner et al., 2009), we focus only on ‘ordinary’ persistence in this paper because of the lower computational complexity.
Persistent homology groups.
The filtration above is connected by the inclusion homomorphism between K ( i ) ⊆
K ( i +1) .
The respective boundary homomorphisms induce a homomorphism between corresponding homology groups of the filtration, i.e. a map i ( i,j ) d : H d (K i )
→ H d (K j ) .
This family of homomorphisms gives rise to a sequence of homology groups
[Equation]
(3) for every dimension d .
For i ≤ j ,
the d th persistent homology group is defined as
[Equation]
Intuitively, this group contains all homology classes created in K ( i ) that are also present in K ( j ) .
Similar to the ‘ordinary’ Betti number, we may now define the d th persistent Betti number as the rank of this group, i.e.
[Equation]
Noting the set of indices i, j , we can see persistent homology as generating a sequence of Betti numbers, as opposed to just calculating a single number.
This makes it possible for us to describe topological features in more detail, and summarise them in a persistence diagram .
Persistence diagrams.
Given a filtration induced by a function f : K → R as described above, we store each tuple ( f ( i ) , f ( j ) ) with multiplicity
[Equation]
in the d th persistence diagram D d . Notice that for most pairs of indices, µ ( d ) i,j = 0 .
Given a point ( x
, y ) ∈ D d , we refer to the quantity pers( x, y ) :
= | y − x | as its persistence .
The idea of persistence arose in multiple contexts (Barannikov, 1994; Edelsbrunner et al., 2002; Verri et al., 1993), but it is nowadays commonly used to analyse functions on manifolds, where high persistence is seen to correspond to features of the function, while low persistence is typically considered noise .
A. E XAMPLES OF G RAPH F ILTRATIONS
Before discussing the computational complexity of persistent homology, we briefly provide some worked examples that highlight the impact of choosing different filtrations for analysing graphs.
Filtrations are most conveniently thought of as arising from a function f : G → R , which assigns a scalar-valued function value to each node and edge of the graph by means of setting f ( u, v ) :
= max { f ( u ) , f ( v ) } for an edge ( u, v ) .
In this context, f is often picked to measure certain salient vertex features of G , such as the degree (Hofer et al., 2017), or its structural role in terms of a heat kernel (Carri`ere et al., 2020).
The resulting topological features are then assigned the respective function values, i.e. ( i, j ) (cid:55)→ ( f ( u i ) , f ( u j )) .
As a brief example, consider a degree-based filtration of a simple graph.
The filtration values are shown as numbers next to a vertex; we use a ( j ) to denote the filtration value at step j (this notation will be useful later when dealing with multiple filtrations).
In each filtration step, the new nodes and edges added to the graph are shown in red, while black elements indicate the structure of the graph that already exists at this step.
[Figure]
[Table] [Table]
[Equation]
[Table] [Table]
[Equation]
[Table] [Table]
Since all edges are inserted at a (3) = 3 , we obtain the following 0 -dimensional persistence diagram
D
= { (1 , ∞ ) , (1 , 3) , (2 , 3) , (3 , 3) , (3 , 3) } .
The existence of a single tuple of the form ( · , ∞ ) indicates that β 0 = 1 .
Similarly, there is only one cycle in the data set, which is created at a (3) , leading to
D 1 = { (3 , ∞ ) } and β 1 = 1 .
If we change the filtration such that each vertex has a unique filtration value, we obtain a different ordering and different persistence tuples, as well as more filtration steps:
[Equation]
Here, connected components are not all created ‘at once’, but more gradually, leading to D 0 = { (1 , ∞ ) , (3 , 3) , (2 , 4) , (4 , 4) , (5 , 5) } .
Of particular interest is the tuple (2 , 4) .
It was created by the vertex with filtration value 2 .
In G (3) it merges with another connected component, namely the one created by the vertex with function value 3 .
This leads to the tuple (3 , 3) , because in each merge, we merge from the ‘younger’ component (the one arising later in the filtration) to the ‘older’ component (the arising earlier in the filtration).
Again, there is only a single cycle in the data set, which we detect at a (5) = 5 , leading to D 1 = { (5 , ∞ ) } .
A. C OMPUTATIONAL D ETAILS
The cardinality of the persistence diagram of dimension-0, D 0 is equal to the number of vertices, n in the graph.
A natural pairing of persistence tuples then consists in assigning each tuple to the node that generated it.
As for dimension-1, D 1 contains as many tuples as cycles in the graph.
However, there is no bijective mapping between dimension-1 persistence tuples and vertices.
Rather, we link each dimension-1 tuple to the edge that created that particular cycle.
To account for multiple distinct filtrations and because the same cycle can be assigned to different edges depending on the specific filtration function, we define a dummy tuple for edges that are not linked to any cycle for a particular filtration.
The set of persistence diagrams ( D ( l ) 1 , . . .
, D ( l ) k ) can then be concatenated as a matrix in
R d l × 2 · k , with d l = | V | (the number of vertices in graph G )
if l = 0 and d l = | E | (the number of edges in G )
if l = 1 .
Remarkably, that leads to Ψ ( l ) (cid:16) D ( l ) 1 , . . .
, D ( l ) k (cid:17) being an operator on a matrix, significantly facilitating computations.
Because of its natural indexing to the vertices, Ψ (0) (cid:16) D (0)1 , . . .
, D (0) k (cid:17) can be mapped back to the graph as explained in Section 4.
For l = 1 , we pool Ψ (1) (cid:16) D (1)1 , . . .
, D (1) k (cid:17) to a graph-level embedding, and mask out the edge indices that are not assigned a persistence pair in any of the D (1) k .
B T HEORETICAL E XPRESSIVITY OF TOGL: P ROOFS
This section provides more details concerning the proofs about the expressivity of our method.
We first state a ‘weak’ variant of our theorem, in which injectivity of the filtration function is not guaranteed.
Next, we show that the filtration function f constructed in this theorem can be used to prove the existence of an injective function ˜ f
that is arbitrarily close to f and provides the same capabilities of distinguishing graphs.
Theorem 3 (Expressivity, weak version) .
Persistent homology is at least as expressive as WL[ 1 ], i.e. if the WL[ 1 ] label sequences for two graphs G and G ′ diverge, there is a filtration f such that the corresponding 0 -dimensional persistence diagrams D 0 and D ′ 0 are not equal.
Proof.
Assume that the label sequences of G and G ′ diverge at iteration h .
Thus, ϕ ( h ) G ̸ = ϕ ( h ) G ′
and there exists at least one label whose count is different.
Let L ( h ) :
= { l 1 , l 2 , . . .
} be an enumeration of the finitely many hashed labels at iteration h .
We can build a filtration function f by assigning a vertex v with label l i to its index, i.e. f ( v ) :
= i , and setting f ( v, w ) :
= max { f ( v ) , f ( w ) } for an edge ( v, w ) .
The resulting 0-dimensional persistence diagram D 0 (and D ′ 0 for G ′ ) will contain tuples of the form ( i, j ) , and each vertex is guaranteed to give rise to exactly one such pair.
Letting µ ( i,j )( D 0 ) refer to the multiplicity of a tuple in D 0 , we know that, since the label count is different, there is at least one tuple ( k, l ) with µ ( k,l )( D 0 ) ̸
= µ ( k,l )( D ′ 0 ) .
Hence, D 0 ̸ = D ′ 0 .
While Theorem 3 proves the existence of a filtration, the proof is constructive and relies on the existence of the WL[ 1 ] labels.
Moreover, the resulting filtration is typically not injective , thus precluding the applicability of Theorem 1.
The following Lemma discusses a potential fix for filtration functions of arbitrary dimensionality; our context is a special case of this.
Lemma 1.
For all ϵ > 0 and f : V → R d there is an injective function ˜ f
such that ∥ f − ˜ f
∥ ∞ < ϵ .
Proof.
Let V = { v 1 , . .
. , v n } the vertices of a graph and im f = { u 1 , . .
, u m } ⊂ R d their images under f .
Since f is not injective, we have m < n .
We resolve non-injective vertex pairs iteratively.
For u ∈ im f , let V ′ :
= { v ∈ V | f ( v ) = u } .
If V ′ only contains a single element, we do not have to do anything.
Otherwise, for each v ′ ∈ V ′ , pick a new value from B ϵ ( u ) \ im f , where
B r ( x ) ⊂ R d refers to the open ball of radius r around a point x (for our case, i.e. d = 1 , this becomes an open interval in R ).
Since we only ever remove a finite number of points, such a new value always exists, and we can modify im f accordingly.
The number of vertex pairs for which f is non-injective decreases by at least one in every iteration, hence after a finite number of iterations, we have modified f to obtain ˜ f , an injective approximation to f .
By always picking new values from balls of radius ϵ , we ensure that ∥ f − ˜ f
∥ ∞ < ϵ , as required.
Using this Lemma, we can ensure that the function f in Theorem 3 is injective, thus ensuring differentiability according to Theorem 1.
However, we need to make sure that the main message of
Theorem 2 still holds, i.e. the two graphs G and G ′ must lead to different persistence diagrams even under the injective filtration function ˜ f .
This is addressed by the following lemma, which essentially states that moving from f to an injective ˜ f does not result in coinciding persistence diagrams.
Lemma 2.
Let G and G ′ be two graphs whose 0 -dimensional persistence diagrams D 0 and D ′ 0 are calculated using a filtration function f as described in Theorem 3.
Moreover, given ϵ > 0 , let ˜ f be an injective filtration function with ∥ f − ˜ f
∥ ∞ and corresponding 0 -dimensional persistence diagrams (cid:102) D 0 and (cid:102) D ′ 0 .
If D 0 ̸ = D ′ 0 , we also have (cid:102) D 0 ̸
= (cid:102) D ′ 0 .
Proof.
Since ˜ f is injective, each tuple in (cid:102) D 0 and (cid:102)
D ′ 0 has multiplicity 1 .
But under f , there were differences in multiplicity for at least one tuple ( k, l ) .
Hence, given ˜ f , there exists at least one tuple ( k, l ) ∈ (cid:102) D 0 ∪ (cid:102) D ′ 0 with ( k, l ) / ∈ (cid:102)
D 0 ∩ (cid:102) D
As a consequence, (cid:102) D
̸ = (cid:102) D ′ 0 .
For readers familiar with TDA, this lemma can also be proved in a simpler way by noting that if the bottleneck distance between D 0 and D ′ 0 is non-zero, it will remain so if one picks a perturbation that is sufficiently small (this fact can also be proved more rigorously).
In any case, the preceding proofs enable us to finally state a stronger variant of the expressivity theorem, initially stated as Theoremin the main paper.
For consistency reasons with the nomenclature in this section, we slightly renamed the filtration function.
Theorem 4 (Expressivity, strong version) .
Persistent homology is at least as expressive as WL[ 1 ], i.e. if the WL[ 1 ] label sequences for two graphs G and G ′ diverge, there exists an injective filtration ˜ f such that the corresponding 0 -dimensional persistence diagrams (cid:102) D 0 and (cid:102)
D ′ 0 are not equal.
Proof.
We obtain a non-injective filtration function f from Theorem 3, under which D 0 and D ′ 0 are not equal.
By Lemma 1, for every ϵ > 0 , we can find an injective function ˜ f with ∥ f − ˜ f
∥ ∞ < ϵ .
According to Lemma 2, the persistence diagrams calculated by this function do not coincide, i.e.
(cid:102) D
̸ = (cid:102) D ′ 0 .
Hence, ˜ f is a filtration function and, according to Theorem 1, differentiability is ensured.
C E MPIRICAL E XPRESSIVITY : A NALYSIS OF (S TRONGLY ) R EGULAR G RAPHS
A k -regular graph is a graph G =
( V, E ) in which all vertices have degree k .
For k = { 3 , 4 } , such graphs are also known as cubic and quartic graphs, respectively.
The Weisfeiler–Lehman test is capable of distinguishing between certain variants of these graphs (even though we observe that
WL[ 1 ] is not sufficient to do so).
Similarly, a strongly regular graph is a graph G =
( V, E ) with two integers λ, µ ∈ N such that each pair of adjacent vertices has λ common neighbours, whereas every pair of non-adjacent vertices has µ common neighbours.
Persistent homology can make use of higher-order connectivity information to distinguish between these data sets.
To demonstrate this, we use a standard degree filtration and compute persistent homology of the graph, including all higher-order cliques.
We then calculate the total persistence of each persistence diagram D , and use it to assign a feature vector to the graph.
This is in some sense the simplest way of employing persistent homology; notice that we are not learning a new filtration but keep a fixed one.
Even in this scenario, we find that there is always a significant number of pairs of graphs whose feature vectors do not coincide—or, conversely speaking, as Table S shows, there are only between 14%–22% of pairs of graphs that we cannot distinguish by this simple scheme.
This illustrates the general expressivity that a topology-based perspective can yield.
For the strongly-regular graphs, we observe even lower error rates: we only fail to distinguish about 1.2% of all pairs (specifically, 8236 out of 7424731 pairs) of the 3854 strongly-regular graphs on 35 vertices with λ = µ = 9 (McKay and Spence, 2001).
A note on computational complexity.
At the same time, this approach is also not without its disadvantages.
Since we perform a clique counting operation, the complexity of the calculation increases considerably, and we would not suggest to use persistent homology of arbitrary dimensions in practice.
While there are some algorithmic improvements in topological constructions (Zomorodian, 2010), na¨ıve persistent homology calculations of arbitrary order may quickly become infeasible for larger data sets.
This can be alleviated to a certain extent by approximating persistent homology (Cavanna et al.
, 2015a;b; Sheehy, 2013), but the practical benefits of this are unclear.
Nevertheless, this experiment should therefore be considered as an indication of the utility of topological features in general to complement and enhance existing architectures.
D E XPERIMENTAL S ETUP & C OMPUTATIONAL R ESOURCES
Following the setup of Dwivedi et al.
(2020), we implemented the following training procedure: All models are initialised with an initial learning rate lr init , which varies between the data sets.
During
[Table] [Table]
training the loss on the validation split is monitored and the learning rate is halved if the validation loss does not improve over a period of lr patience .
Runs are stopped when the learning rate gets reduced to a value of lower than lr min .
The parameters for the different data sets are shown in
Table S
[Table]
Our method is implemented in Python , making heavy use of the pytorch-geometric library (Fey and Lenssen, 2019), licensed under the MIT License, and the pytorch-lightning library (Falcon et al., 2019), licensed under the Apache 2.0 License.
The training and hyperparameter selection was performed using ‘Weights and Biases’ (Biewald, 2020), resulting in additional reports, tables
[Table]
and log information, which will simplify the reproducibility of this work.
We will make our own code available, using either the MIT License or a BSD 3-Clause license, which precludes endorsing/promoting our work without prior approval.
All licenses used for the code are compatible with this licensing choice.
As for the data sets, we use data sets that are available in pytorch-geometric for graph learning tasks.
Some of the benchmark data sets have been originally provided by Morris et al.
(2020), others (CIFAR-10, C LUSTER , MNIST, P ATTERN ) have been provided by Dwivedi et al.
(2020) in the context of a large-scale graph neural network benchmarking effort.
Ablated static variant of TOGL.
Next to all the experiments presented in the main paper, we also developed a static variant of our layer, serving as an additional ablation method (this nomenclature will be used in all supplementary tables).
The static variant mimics the calculation of a filtration in terms of the number of parameters but without taking topological features into account.
The layer uses a static mapping (instead of a dynamic one based on persistent homology) of vertices to themselves (for dimension 0 ), and employs a random edge selection process (for dimension 1 ).
This has the effect of learning graph-level information that is not strictly based on topology but on node feature values.
The static variant of TOGL reduces to an MLP that is applied per vertex in case interactions between tuples are not considered, and to the application of a DeepSet Zaheer et al.
(2017) to the vertex representations if interactions between tuples are incorporated.
Generally, if the static variant performs well on a data set, we assume that performance is driven much more by the availability of any graph-level type of information, such as the existence of certain nodes or groups of nodes, as opposed to topological information.
Compute resources.
Most of the jobs were run on our internal cluster, comprising 64 physical cores ( Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz ) with 8 GeForce GTX
GPUs.
A smaller fraction has been run on another cluster, containing 40 physical cores ( Intel(R)
[Table] [Table]
GPU.
E H YPERPARAMETERS
Table S6 contains a listing of all hyperparameters used to train TOGL.
For the Weisfeiler–Lehman subtree kernel, which we employed as a comparison partner on the synthetic data sets, we used a support vector machine classifier with a linear kernel
[Table]
whose regularisation parameter
C ∈ { 10 − 4 , 10 − 3 , . .
, 10 4 } is trained via 5 -fold cross validation, which is repeated 10 times to obtain standard deviations.
This follows closely the setup in graph classification literature.
F E XTENDED R ESULTS FOR THE S YNTHETIC D ATA S ETS
In the main paper
[Table]
we depicted a concise analysis of synthetic data sets (Figure 1).
Here, we provide a more detailed ablation of this data set, highlighting the differences between TOGL and its static baseline
[Table]
but also disentangling the performance of 0 -dimensional and 1 -dimensional topological features, i.e. connected components and cycles, respectively.
Figure S4 depicts the results.
As stated in the main paper, we observe that (i) both cycles and connected components are crucial for reaching high predictive performance, and (ii) the static variant is performing slightly better than the standard
GCN, but significantly worse than TOGL, due to its very limited access to topological information.
G V ISUALISATION OF THE L EARNT F ILTRATIONS
TOGL can learn an arbitrary number of filtrations on the graphs.
In Figure S7, we display 3 different filtrations learnt on an randomly picked graph from the DD data set and compare it against the classical node degree filtration.
The width of the nodes is plotted proportional to the node degree filtration while the colour saturation is proportional to the learnt filtration.
Filtration 0 appears to be correlated with the degree filtration, while other filtrations learnt by TOGL seem to focus on different local properties of the graph.
H E XPERIMENTS ON L AYER P LACEMENT
A priori, it is not clear where to place TOGL.
We therefore investigated the impact of putting TOGL first (thus imbuing the graph with topological features arising from the neighbours of nodes) or at another position in the network.
In Table Table S7, we investigate the performance of the layer when
TOGL is placed at different positions in a GNN architecture composed of 3 graph convolution layers.
It appears that for this dataset, positioning TOGL before the last GNN layer leads to best performance
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table] [Table]
when the non-static version is considered.
Importantly, this contrasts with the approach of Hofer et al.
(2019), where topological information is only available at the read-out level, which would lead to the worst performance on this dataset.
We complemented our experiments on layer placement with an exhaustive assessment of the performance of our model on on the IMDB-Binary, Reddit-Binary, Proteins and DD datasets.
Please refer to Table S8 for these results.
I E XTENDED E XPERIMENTS FOR S TRUCTURED -B ASED G RAPH C LASSIFICATION
This section contains more results for the structured-based classification of graph data sets shown in Section 5.3 and Figure 3.
Table Table S9 contains a detailed listing of performance values under
[Table] [Table]
[Table] [Table]
i
[Table] [Table]
[Figure]
certain additional ablation scenarios, such as a disentangled comparison of the performance with respect to picking a certain type of embedding scheme for topological features, i.e. deep sets versus known embedding functions for persistence diagrams, the latter of which cannot handle interactions between tuples in a diagram.
The version labelled General consist of non-static versions where the type of feature embedding scheme is considered as an hyper-parameter.
We then report the test accuracy of the version whose validation accuracy was highest.
The General -Static variant is similar but for the static version of TOGL.
Figure S8 depicts the performance on different data sets, focusing on a comparison between the different embedding types and the static variant of our layer.
The behaviour of our static variant indicates that predictive performance is not driven by having access to topological information on the benchmark data sets.
To some extent, this is surprising, as some of the data sets turn out to contain salient topological features even visual inspection (Figure S5), whereas the social networks exhibit more community-like behaviour (Figure S6).
Together with the results from Section 5.3, this demonstrates that existing data sets often ‘leak’ structural information through their labels/features.
Yet, for the node classification data sets C LUSTER and P ATTERN ,
GCN-3-TOGL-1 or its static variant perform best among all comparison partners.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
J G EOMETRIC D ATASET
In this section
[Table]
we showcase the potential of TOGL for improving performance on tasks of geometric data sets.
Tasks such as 3D object recognition are gathering increasing attention in the machine learning community and our approach represents a promising direction to explore that area.
We consider a simple synthetic data set consisting of graphs linking points on geometrical objects.
In particular, we create a data set with two classes where one consists of graphs lying on a sphere and the other of graphs lying on a torus.
The nodes of these graphs are composed of points sampled on the geometrical objects and embedded in an higher dimensional space.
The graph is then constructed as a fully connected graphs between all sampled points.
Table S12 presents the classification accuracy of different methods on this experiment.
We observe that substituting a GNN layer with a TOGL layer allows to obtain nearly perfect accuracy, showcasing the crucial importance of a topology-informed layer in geometric data sets.
K L ARGER NUMBER OF LAYERS IN THE GNN ARCHITECTURE
In this section, we investigate the performance of our approach.
on architecture with a larger number of layers.
For computational reasons, we limit our analysis to 3 data sets, namely DD, PROTEINS and CIFAR-10.
We re-use the experimental setup as detailed in Section 5.1.
Tables S13 and S show the classification accuracy for the structural and the benchmark setup respectively.
What we observe is that the advantage of including TOGL is generally conserved, and similar to the one observed in the 4 layers case.
In absolute terms, for the structural dataset, the performance on the DD dataset increases with more layers (for both the baseline and the TOGL version) but decreases for the
PROTEINS dataset.
On the benchmark dataset (including node features), the performance tends to increase for all three datasets compared to the 4-layers version.
[Table]