Template based Graph Neural Network with Optimal Transport Distances
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Current Graph Neural Networks (GNN) architectures generally rely on two important components: node features embedding through message passing, and aggregation with a specialized form of pooling.
The structural (or topological) informationis implicitly taken into account in these two steps.
We propose in this work a novelpoint of view, which places distances to some learnable graph templates at thecore of the graph representation.
This distance embedding is constructed thanksto an optimal transport distance: the Fused Gromov-Wasserstein (FGW) distance,which encodes simultaneously feature and structure dissimilarities by solving asoft graph-matching problem.
We postulate that the vector of FGW distances toa set of template graphs has a strong discriminative power, which is then fed toa non-linear classiﬁer for ﬁnal predictions.
Distance embedding can be seen as anew layer, and can leverage on existing message passing techniques to promotesensible feature representations.
Interestingly enough, in our work the optimal setof template graphs is also learnt in an end-to-end fashion by differentiating throughthis layer.
After describing the corresponding learning procedure, we empiricallyvalidate our claim on several synthetic and real life graph classiﬁcation datasets,where our method is competitive or surpasses kernel and GNN state-of-the-artapproaches.
We complete our experiments by an ablation study and a sensitivityanalysis to parameters.
Introduction
Attributed graphs are characterized by i) the relationships between the nodes of the graph (structuralor topological information) and ii) some speciﬁc features or attributes endowing the nodes themselves.
Learning from those data is ubiquitous in many research areas [3], e g. image analysis [20, 8],brain connectivity [29], biological compounds [22] or social networks [67], to name a few.
Variousmethodologies approach the inherent complexity of those data, such as signal processing [52],
Bayesian and kernel methods on graphs [40, 28] or more recently Graph Neural Networks (GNN)[62] in the framework of the geometric deep learning [8, 7].
We are interested in this work in the classiﬁcation of attributed graphs at the instance level .
Oneexisting approach consists in designing kernels that leverage topological properties of the observedgraphs [5, 16, 18, 51].
For instance, the popular Weisfeiler-Lehman (WL) kernel [50] iterativelyaggregates for each node the features of its k -hop neighborhood.
Alternative approaches aim atlearning vectorial representations of the graphs that can encode the graph structure ( i.e. graphrepresentation learning [9]).
In this domain, GNN lead to state-of-the-art performances with endto-end learnable embeddings [62].
At a given layer, these architectures typically learn the nodeembeddings via local permutation-invariant transformations aggregating its neighbour features [36,24, 19, 65].
In order to obtain a representation of the whole graph suitable for classiﬁcation, GNNs
[Table]
BBB@ 0 .
CCCA <
[Table]
C i
[Table]
h
i
[Table] [Table]
[Table] [Table]
Learnable
[Table]
/987cznYWhr6K8PVd0uDtVf2Pi3iCZ7iuUxgjTPZwIgnnuIjPuGz88X55vx0fv2lLnSs5jFmlvP7DyRa3aA=</latexit> operate
Another line of works targets the construction of meaningful distances that integrate simultaneouslythe structural and feature information, and that are based on optimal transport (OT) [59, 45].
Originallydesigned to compare probability distributions based on a geometric notion of optimality, it allowsdeﬁning very general loss functions between various objects, modeled as probability distributions.
Ina nutshell, it proceeds by constructing a coupling between the distributions that minimizes a speciﬁccost .
Some approaches dealing with graphs rely on non-parametric models that ﬁrst embed the graphsinto a vectorial space and then match them via OT [42, 54, 26, 34].
Recently [10] proposed the
OT-GNN model, that embeds a graph as a vector of the Wasserstein distances between the nodes’embeddings (after GNN pre-processing) and learnt point clouds, acting as templates.
Building further from OT variants, the Gromov-Wasserstein (GW) distance [38] directly handlesgraphs through the symmetric matrix C that encodes the distance/similarity between every pairs ofnodes ( e.g. adjacency, shortest path) and a weight vector h on the nodes encoding the nodes’ relativeimportance.
GW has proven to be useful for tasks such as graph matching and partitioning [64, 14]or unsupervised graph dictionary learning [63, 61, 60].
GW has been also extended to directedgraphs [13] and to attributed graphs via the Fused Gromov-Wasserstein (FGW) distance [57, 56],that realizes a trade-off between an OT distance with a cost on node features and the GW distancebetween the similarity matrices.
Despite its recent successes on complex unsupervised tasks suchas graph clustering [63, 61, 60], FGW has never been explored as part of an end-to-end model forgraph classiﬁcation.
In this work, we ﬁll this gap by introducing a novel “layer” that embeds anattributed graph into a vector, whose coordinates are FGW distances to few (learned) graph templates.
While FGW can be performed directly on raw data ( i.e. the input structured graph without anypre-processing), we also consider the case where features representations are learnt from a GNN,similarly to OT-GNN [10], and thus also realizing a particular type of aggregation.
Contributions.
We introduce a new GNN layer, named TFGW for Template-based FGW andillustrated in the center of Figure 1.
From an input graph, it computes a vector of FGW distances tolearnable graph templates.
This layer that can be seen as an alternative to global pooling layers andcan be integrated into any neural network architecture.
We discuss its properties and the associatedinvariances.
We detail the optimization strategy that enables learning simultaneously GNN preprocessing layers and graph templates relevant for a downstream task in an end-to-end fashion.
Weempirically demonstrate the relevance of our model in terms of performances compared to severalstate-of-the-art architectures.
Remarkably, we show that a simple GNN model leveraging on our newlayer can surpass state-of-the-art performances by a relatively large margin.
Finally, we also providesome illustrative interpretations of our method and a sensitivity analysis of our model parameters.
Fused Gromov-Wasserstein template based layer
In order to describe our novel template-based GNN layer
we ﬁrst introduce more formally the
FGW distance and its properties.
In the following we denote by Σ n := { h ∈ R n + | (cid:80) i h i
= 1 } theprobability simplex with n -bins, and by S n ( A ) the set of symmetric matrices of size n taking valuesin A ⊂ R .
2.1 Fused Gromov-Wasserstein distance
An undirected attributed graph G with n nodes can be modeled in the OT context as a tuple ( C , F , h ) , where C ∈ S n ( R ) is a matrix encoding relationships between nodes, F = ( f 1 , ..., f n ) (cid:62) ∈ R n × d isa node feature matrix and h ∈ Σ n is a vector of weights modeling the relative importance of thenodes within the graph (Figure 1, left).
Without any prior knowledge, uniform weights can be chosen( h = 1 n /n ).
The matrix C can be the graph adjacency matrix, the shortest-path matrix or any otherdescription of the node relationships (i.e. the topology) of the graph [46, 56, 14].
Let us now considertwo such graphs ( C , F , h ) and ( C , F , h ) , of respective sizes n and n (with possibly n (cid:54) = n ).
The
Fused Gromov-Wasserstein ( FGW ) distance is deﬁned for α ∈ [0 , 1] as [56, 57]:
[Equation]
where U ( h , h ) :
= { T ∈ R n × n + | T 1 n = h , T (cid:62) 1 n = h } is the set of admissible coupling between hand h .
FGW aims at ﬁnding an optimal coupling T (cid:63) by minimizing a trade-off cost, via α , between a
Wasserstein (W) cost on the features and a Gromov-Wasserstein (GW) cost on the similarity matrices,both sharing the same coupling.
The optimal coupling T (cid:63) acts as a soft matching of the nodes, whichtends to associate pairs of nodes that have similar pairwise relations in C and C (GW cost), andsimilar features in F and F (W cost).
Interestingly, FGW deﬁnes a metric on the space of attributed graphs.
In particular, if C and Care shortest-path matrices, the FGW distance vanishes if and only if the two attributed graphs arethe same up to a permutation [57, Theorem 3.2].
Such invariance involves that two graphs stronglyisomorphic according to Weisfeiler-Lehman base tests [32, 54] will have a zero FGW distance forany α and, more importantly,
FGW α = 0 implies that the graphs are strongly isomorphic 1 .
When
C , C are any symmetric matrices, we can mention that GW ( α = 1 ) also deﬁnes a pseudo-distance[53, Theorem 5.8] with respect to the notion of weak isomorphism [53, 14].
Solving for FGW.
The optimization problem 1 is a non-convex quadratic program [57, equation6], whose non-convexity comes from the GW cost.
A possible optimization procedure to solve thisproblem is a Conditional Gradient (CG) algorithm, which is known to converge to a local optimum[30].
The computational complexity of each iteration is O ( n 2 n + n 2 n ) [46].
Thus, if two graphs ofconsiderably different sizes are considered, the complexity is quadratic with respect to the largestsize.
Existing attempts to reduce this computational cost either exploit entropic regularization of OT[46, 48] or graph partitioning [64, 12].
2.2 Template-based (T)FGW Graph Neural Networks
Building upon the FGW distance and its properties, we propose a simple layer for a GNN thattakes a graph ( C , F , h ) as input and computes its FGW distances to a list of K template graphs
G :=
{ ( C k , F k , h k ) } k ∈
[[ K ]] as follows :
[Equation]
We postulate that this graph representation can be discriminant between the observed graphs due to
FGW .
This claim relies on the theory of [2] allowing one to learn provably strongly discriminantclassiﬁers based on the distances from the observed graphs and templates that are sampled from thedataset (see e.g. [47] adopting the Wasserstein distance).
However such an approach often requires a large amount of templates which might be prohibitive if the distance is costly to compute.
Instead,we propose to learn the graph templates G in a supervised manner.
In the same way, we also learnthe trade-off parameter α on the data.
As such, the TFGW layer can automatically adapt to thedata whose discriminating information can be discovered either in the features or in the structure ofthe graphs, or in a combination of the two.
Moreover, the template structures can leverage on anytype of input representation C i since they are learnt directly from the data.
Indeed, in the numericalexperiments we implemented the model using either adjacency matrices (ADJ) that provide moreinterpretable templates (component C i,j ∈ [0 , 1] can be seen as a probability of link between nodes)or shortest path matrices (SP) that are more complex to interpret but encode global relations betweenthe nodes.
The TFGW layer can be used directly as a ﬁrst layer to build a graph representation feeding afully connected network (MLP) for e.g. graphs classication.
In order to enhance the discriminatingpower of the model, we propose to put a GNN (denoted by φ u and parametrized by u ) on top of the
TFGW layer.
We assume in the remainder that this GNN model φ u is injective in order to preserveisomorphism relations between graphs (see [65] for more details).
With a slight abuse of notation, wewrite φ u ( F ) to denote the feature matrix of an observed graph after being processed by the GNN.
Learning with TFGW-GNN.
We focus on a classiﬁcation task where we observe a dataset D of
I graphs {G i =
( C i , F i , h i ) } i ∈
[[ I ]] with variable number of nodes { n i } i ∈
[[ I ]] and where each graph 131 is assigned to a label y i ∈ Y , with Y a ﬁnite set.
The full model is illustrated in Figure 1.
Weﬁrst process the features of the nodes of the input graphs via the GNN φ u , then use the TFGWlayer to represent the graphs as vectors in R K .
Finally we use the ﬁnal MLP model ψ v : R K
→ Yparameterized by v , to predict the label for any input graph.
The whole model is learned in aend-to-end fashion by minimizing the cross-entropy loss on the whole dataset leading to the followingoptimization problem :
[Equation]
Notable parameters of (3) are the template graphs in the embeddings { ( C k , F k , h k ) } and moreprecisely their pairwise node relationship C k , node features F k and the distribution on the nodes h kon the simplex.
The last parameter reweighs individual nodes in each template and performs nodesselection when some weights are exactly 0 [61, 60].
Finally, the global parameter α is also learntfrom the whole dataset.
Although it is possible to learn a different α per template, we observed thatthis extra level of ﬂexibility is prone to overﬁtting, and we will not consider it in the experimentalsection.
Optimization and differentiation of TFGW.
We propose to solve the optimization problem in(3) using stochastic gradient descent.
The FGW distances are computed by adapting the conditionalgradient solver implemented in the POT toolbox [17].
The solver was designed to allow backwardpropagation of the gradients w.r.t.
all the parameters of the distance and was adapted to also computethe gradient w.r.t.
the parameter α .
The gradients are obtained using the Envelop Theorem [1]allowing to keep T (cid:63) constant.
We used Pytorch [44] to implement the model.
The template structure
C k , node weights h k and α are updated with a projected gradient respectively on the set of symmetricmatrices S n k
( R + ) ( S n k ([0 , 1]) when C i are adjacency matrices), the simplex Σ n k and [0 , 1] .
Theprojection onto the probability simplex of the node weights leads to sparse solutions [15], thereforethe size of each ( C k , F k , h k ) can decrease along iterations hence reducing the effective numberof their parameters to optimize.
This way the numerical solver can leverage on the fact that manycomputations are unnecessary as soon as the weights are set to zero.
Note that the FGW solver from
POT uses an OT solver implemented in C++ on CPU which means that it comes with some overhead(memory transfer between GPU and CPU) when training the model on GPU.
Still the multiple FGWdistances computation has been implemented in parallel on CPU with a computational time thatremains reasonable in practice (see experimental section 3.3).
While a GPU solver can be foundwhen using entropy regularized FGW, it introduces a new parameter related to the regularizationstrength which is more cumbersome to set, and that we did not consider it in the experiments.
Properties of the TFGW layer.
We now discuss a property of the proposed layer resulting fromthe properties of FGW (see Section 2.1).
We have the following result:
Lemma
The TFGW embeddings are invariant to strong isomorphism.
This lemma directly stems from the fact that FGW is invariant to strong isomorphism of one ofits inputs.
This proposition implies that two graphs with any aforementioned representation whichonly differ by a permutation of the nodes will share the same TFGW embedding.
Moreover such aproperty holds for any mapping φ u which is injective, such as a Multi-Layer Perceptron (MLP)
[21]or any GNN with a sum aggregation scheme as described in [65].
Moreover, the optimal coupling T (cid:63) resulting from TFGW between ( C i , φ u ( F i ) , h i ) and the template ( C k , F k , h k ) , will encode correspondances between the nodes of the graph and the nodes ofthe template that will be propagated during the backward operation.
The size of the inputs, the size ofthe templates and their respective weight h k will play a crucial role regarding this operation.
Alsonote that, since the templates are estimated here to optimize a supervised task, they will promotediscriminant distance embedding instead of graph reconstruction quality as proposed in other FGWunsupervised learning methods [61, 60].
Numerical experiments
This section aims at illustrating the performances of our approach for graph classiﬁcation in syntheticand real-world datasets.
First, we showcase the relevance of our TFGW layer on existing syntheticdatasets known to require expressiveness beyond the WL-test (Section 3.1).
Then we benchmark ourmodel with state-of-the-art approaches on well-known real-world datasets (Section 3.2).
We ﬁnallydiscuss our results through a sensitivity analysis of our models (Section 3.3).
3.1 Synthetic datasets beyond WL test
[Table]
[Figure]
Identiﬁcation of graphs beyond the WL test is one important challenge faced by the GNN community.
In order totest the ability of TFGW to handle such fundamentallydifﬁcult problems we consider two synthetic datasets: 4CYCLES [33, 43] contains graphs with (possibly) disconnected cycles where the label y
i is the presence of a cycleof length 4; SKIP-CIRCLES [11] contains circular graphswith skip links and the labels (10 classes) are the lengthsof the skip links among { 2 , 3 , 4 , 5 , 6 , 9 , 11 , 12 , 13 , 16 } .
We compare the performances of the TFGW layer for embedding such graphs with GIN [65] designedto be at least as expressive as the WL test, and DropGIN [43] which proposed a successful dropouttechnique to overcome some drawbacks of GIN.
We replicate the benchmark of [43] by consideringfor both GIN and DropGIN, 4 GIN layers for 4-CYCLES, and 9 GIN layers for SKIP-CIRCLES asthe skip links can form cycles of up to 17 hops.
Since the graphs do not have features we use directlythe TFGW on the raw graph representation with α = 1 hence computing only the GW distance.
[Figure]
The GNN methods above artiﬁcially adds a feature equal to 1 on allnodes as they have the same degree.
For these experiments we useadjacency matrices for C
i
and we investigate two ﬂavours of TFGW :
1)in TFGW -ﬁx we ﬁx the templates by sampling one template per classfrom the training dataset (this can be seen as a simpler FGW featureextraction); 2) for TFGW we learn the templates from the training data(as many as the number of classes) as proposed in the previous sections.
Results are averaged over 10 runs and reported in Table 1.
TFGW basedmethods perform very well on both datasets with impressive results on
SKIP-CIRCLE when GNN have limited performances.
This is due to thefact that different samples from one class of SKIP-CIRCLE are generatedby permuting nodes of the same graph and FGW distances are invariantto these permutations.
4-CYCLES has a more complex structure withintra-class heterogeneity and requires more than two templates to performas good as DropGIN.
To illustrate this we have computed the accuracyon this dataset as a function of the number of templates K in Figure 2.
We can see that a perfectclassiﬁcation is reached up to K = 4 for TFGW , while TFGW -ﬁx still struggles to generalize at
[Table]
K = 20 .
This illustrates that learning the templates is essential to keep K (and numerical complexity)small while ensuring good performances.
3.2 Graph classiﬁcation benchmmark
We now evaluate and compare the performances of our TFGW GNN with a number of state-of-the-artgraph classiﬁers, from kernel methods to GNN.
The numerical experiments are conducted on real lifegraph datasets to provide a fair benchmark of all methods on several heterogeneous graph structures.
Datasets.
We use 8 well-known graph classiﬁcation datasets [23]: 5 bioinformatics datasets amongwhich 3 have discrete node features (MUTAG, PTC, NCI1 [27, 50]) and 2 have continuous nodefeatures (ENZYMES, PROTEINS[6]) and 3 social network datasets (COLLAB, IMDB-B, IDBM-M[67]).
In order to analyse them with all methods, we augment unattributed graphs from socialnetworks with node degree features.
Detailed description and statistics on these datasets are reportedin the supplementary material.
Baselines.
We benchmark our approaches to the following state-of-the-art baselines for graphsclassiﬁcation, split into 3 categories: i) kernel based approaches , including FGW [57] operating onadjacency and shortest-path matrices, the WL subtree kernel [50, WL] and the Wasserstein WL kernel[54, WWL].
For these methods that do not require a stopping criterion dependent on a validation set,we report results using for parameter validation a 10-fold nested cross-validation [57, 28] repeated 10times.
ii)
OT based representation learning models, including WEGL [26] and OT-GNN [10].
iii)
GNN models , with global or more sophisticated pooling operations, including PATCHY-SAN [41],
DIFFPOOL [68], PPGN [35], GIN [65] and its augmented version through structure perturbations
DropGIN [43].
For all these methods, we adopt the hyper-parameters suggested in the respectivepapers, but with a slightly different model selection scheme, as detailed in the next paragraph.
Benchmark settings.
Recent GNN literature [36, 65, 35, 43] successfully addressed many limitations in terms of model expressiveness compared to the WL tests.
Within that scope, they suggestedto benchmark their models using a 10-fold cross-validation (CV) where the best average accuracyon the validation folds was reported.
We suggest here to quantify the generalization capacities of
GNN based models by performing a 10-fold cross validation with a holdout test set never seen duringtraining.
For each split, we track the accuracy on the validation fold every 5 epochs, then the modelwhose parameters maximize that accuracy is retained.
Finally, the model used to predict on theholdout test set is the one with maximal validation accuracy averaged across all folds.
This settingis more realistic than a simple 10-fold CV and allows a better understanding of the generalizationperformances [4].
This point explains why some existing approaches have here different performancesthan those reported in their original paper.
For all the TFGW based approaches we empirically study the impact of the input structure representation by considering adjacency (ADJ) and shortest-path (SP) matrices C i .
For all template basedmodels, we set the size of the templates to the median size of the observed graphs.
We validate the number of templates K in { β |Y|} β , with β ∈ { 2 , 4 , 6 , 8 } and |Y| the number ofclasses.
Only for ENZYMES with 6 classes of 100 graphs each, we validate β ∈ { 1 , 2 , 3 , 4 } .
Allparameters of our TFGW layers highlighted in red in Figure 1 are learned while φ u is a GINarchitecture [65] composed of L = 2 layers aggregated using the Jumping Knowledge scheme [66]known to prevent overﬁtting in global pooling frameworks.
For OT-GNN we validate the numberof GIN layers in L ∈ { 2 , 4 } .
Finally for fairness, we validate the number of hidden units within the
GNN layers and the application of dropout on the ﬁnal MLP for predictions, similarly to GIN and
DropGIN.
Results analysis.
The results of the comparisons in terms of accuracy are reported in Table 2.
Our
TFGW approach consistently outperforms with signiﬁcant margins the state-of-the-art approachesfrom all categories.
Even if most of the benchmarked models can perfectly ﬁt the train sets by learningimplicitly the graphs structure [65, 43, 35], enforcing such knowledge explicitly as our TFGW layerdoes (through FGW distances) leads to considerably stronger generalization performances.
On 7 outof 8 datasets, TFGW leads to better performances while operating on adjacency matrices (TFGW
ADJ) than on shortest-path ones (TFGW SP).
Interestingly, this ranking with respect to those input representations does not necessarily match the one of the FGW kernel which extracts knowledge from thegraph structures
C i through FGW , as our TFGW layer.
These different dependencies to the providedinputs may be due to the GNN pre-processing of node features which suggests the study of its ablation.
[Table]
Finally to complete this analysis, we report in Table 3the number of parameters of best selected models acrossvarious methods, for the dataset PTC.
We refer the readerinterested in an extended benchmark to the supplementary material.
Our TFGW leads to better classiﬁcationperformances while having comparable number of parameters than these competitors.
We also reported for thesemodels, their averaged prediction time per graph.
Thesemeasures were taken on CPUs (Intel Core i9-9900K CPU,3.60 GHz) in order to fairly compare the numerical complexity of these methods, as OT solver used in TFGW and OT-GNN are currently limited to thesedevices (see the detailed discussion in Section 2.1).
Although the theoretical complexity of ourapproach is at most cubic in the number of nodes, we still get in practice a fairly good speed forclassifying graphs, in comparison to the other competitive methods.
3.3 Ablation study, sensitivity analysis and discussions
In this section we inspect the role of some of the model parameters ( α in FGW , weights estimationin the templates, depth of the GNN φ u ) in terms of the classiﬁcation performance.
To this end, weﬁrst conduct on all datasets an ablation study on the graph template weights h k and the number of
GIN layers in φ u .
Then, we take a closer look at the estimated trade-off parameters α and provide asensitivity analysis w.r.t.
the number of templates and the number of GIN layers.
Ablation Study.
Following the same procedure as in Section 3.2, we benchmark the followingsettings for our TFGW models: for adjacency (ADJ) and shortest path (SP) representations C i , welearn the distance layers either directly, on the raw data ( i.e. L = 0 , φ u = id ), or after embedding thedata with ( L = 1) or ( L = 2) GIN layers.
For L =
we either ﬁx the graph template weights h kuniformly or learn them.
The results (test accuracy) are reported in Table 4.
Learning the weightssystematically improves the generalization capabilities of our models of at least 1% or 2% for both
ADJ and SP graph representations.
For a given number of graph templates, the weights learningallows to better ﬁt the speciﬁcities of the classes (e.g. varying proportion of nodes in different parts ofthe graphs).
Moreover, as weights can become sparse in the simplex during training they also allowthe model to have templates whose number of nodes adapts to the classiﬁcation objective, whilebringing computational beneﬁts as discussed in Section 2.2.
Those observations explain why welearnt them by default in the benchmark of the previous subsection.
Next, we see in Table 4 that using GNN layers as a pre-processing for our TFGW layer enhancesgeneralization powers of our models, whose best performances are obtained for L = 2 .
Interestingly,for L = 0 , TFGW with SP matrices outperforms TFGW with ADJ matrices, meaning that theshortest path distance brings more discriminant information on raw data.
But when L ≥ 1 ( i.e. when
[Table]
81.1(2.5) TFGW (L=2)
ADJ learnt 96.4(3.3) 72.4(5.7) 73.8(4.6) 82.9(2.7) [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Figure]
[Figure]
a GNN pre-processes the node features), TFGW with ADJ matrices improves the accuracy.
Anexplanation could be that the GNN φ u can somehow replicate (and outperform) a SP metric betweennodes.
This emphasizes that the strength of our approach clearly exhibited in Table 2 lies in theinductive bias of our FGW distance embedding.
This last point is further reinforced by additionalexperiments reported in the supplementary material, where GIN layers (used by default for our TFGWmodel) are replaced by GAT layers [58].
[Figure]
Importance of the structure/feature aspect of
FGW.
To the best of our knowledge we are theﬁrst to actually learn the trade-off parameter αof FGW in a supervised way.
For this matter,we verify that our models did not converge todegenerated solutions where either the structure( α = 0 ) or the features ( α = 1 ) are omitted.
Tothis end we report in Figure 4 the distributionsof the estimated α for some models learnt ondatasets PTC and IMDB-B, where features are respectively existing in the dataset or created usingnode degrees.
We can see that for both kinds of input graph representations, α parameters are strictlybetween 0 and 1 .
One can notice the variances of those distributions illustrating the non-uniquenessof this trade-off parameter coming from the non-convexity of our optimization problem (a given valueof α can potentially be compensated by the scaling of the GNN output).
Unfortunately, the analysisof the real relative importance between structures and features can not be achieved only by looking atthose values as the node embeddings and templates are different across models and data splits.
Visualizing the TFGW embedding.
In order to interpret the TFGW embedding, we illustrate in
Figure 3 the PCA projection of our distance embeddings learned on PTC with L = 0 and L =
2and the number of templates K varying in { 4 , 8 } .
For this experiment, we have chosen the PCAbecause it allows to have a more interpretable low dimensional projection that preserves the geometrycompared to local neighbourhood based embeddings such as TSNE [55] or UMAP [37].
As depictedin the ﬁgure, the learned templates are extreme points in the embedding space of the PCA.
This resultis particularly interesting because existing unsupervised FGW representation learning methods tend toward estimating templates that belong to the data manifold, or to form a “convex enveloppe” ofthe data to ensure good reconstruction [61, 63].
On the contrary, the templates learned through ourapproach seem to be located on a plane in the PCA space while the samples evolve orthogonally tothis plane (when the FGW distance increases).
In a classiﬁcation context, this means that the learnedtemplates will not actually represent realistic graphs from the data but might encode “exaggerated” or“extreme” features in order to maximize the margin between classes in the embedding.
To reinforcethis intuition, we added plots of the estimated templates in the supplementary.
Finally, in the ﬁgure,the samples are coloured w.r.t.
their class and the templates are coloured by their predicted class.
Interestingly, the classes are already well separated with 4 templates but the separation is clearlynon-linear whereas using GNN pre-processing and a larger number of templates leads to a linearseparation of the two classes.
[Figure]
Sensitivity to the number of templates and GNN layers.
To illustrate the sensitivity of our TFGWlayer to the number of templates Kand the number of GNN layers Lin φ u , we learned our models onthe PTC dataset with L varying in{ 0 , 1 , 2 , 3 , 4 } .
We follow the sameprocedure than in the benchmark of
Section 3.2 regarding the validationof K and the learning process, whileﬁxing the number of hidden units inthe GNN layers to 16 .
The test accuracy distributions for all settings arereported in Figure 5.
Two phases areclearly distinguishable.
The ﬁrst one for L ≤ 2 , where for each L we see that the performanceacross the number of templates steadily increases, and the second for L > 2 where this performanceprogressively decreases as a function of L .
Moreover, in the ﬁrst phase performances are considerablydependent on K to compensate for a simple node representation, while this dependency is mitigated inthe second which exhibits a slight overﬁtting.
Note that these deeper models still lead to competitiveresults in comparison with benchmarked approaches in Table 2, with best averaged accuracies of 70 .
6( L = 3) and 67 . 4 ( L = 4) .
On one hand, these observations led us to set our number of layers to
L = 2 for all benchmarked datasets which lead to strong generalization power.
On the other hand,deeper models might be a way to beneﬁt from our FGW embeddings with very few templates whichcan be interesting from a computational perspective on larger graph datasets.
Conclusion
We have introduced a new GNN layer whose goal is to represent a graph by its distances to templategraphs, according to the optimal transport metric FGW .
The proposed layer can be used directly onraw graph data as the ﬁrst layer of a GNN or can also beneﬁt from more involved node embeddingusing classical GNN layers.
In a graph classiﬁcation context, we combined this TFGW layer with asimple MLP model.
We demonstrated on several benchmark datasets that this approach comparedfavorably with state-of-the-art GNN and kernel based classiﬁers.
A sensitivity analysis and an ablationstudy were presented to justify the choice of several parameters explaining the good generalizationperformances.
We believe that the new way to represent complex structured data provided by TFGW will openthe door to novel and hopefully more interpretable GNN architectures.
From a practical perspective,future works will be dedicated to combine TFGW with fast GPU solvers for network ﬂow [49].
Thiswould greatly accelerate our approach and more generally OT based deep learning methods.
We alsobelieve that the FGW distance and its existing extensions can be used with other learning strategiesincluding semi-relaxed FGW [60] for sub-graph detection.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] Every key points are encapuslated in the contributions
part at the end of the introduction.
(b) Did you describe the limitations of your work?
[Yes] Yes the limitations of our work
mostly lie in its computational complexity and the lack of a current implementation
designed for GPU parallelization of the FGW distance solver in section 2.1 and 3.2.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Indeed the full
set of assumptions of all theoretical results are in the paper.
Few technical deﬁnitions
which are common in the OT litterature have been reported to the supplementary
material.
(b) Did you include complete proofs of all theoretical results?
[Yes] A simple intuition for
Lemma 1 is given in the paper and more details are provided in the supplementary.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] The code is
provided in the supplementary material.
A read me has been made to reproduce these
experiments more easily.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] For each kind of experiments most settings are provided in the core
of the paper
(see section 3).
Complementary details on the experiments are reported in
the supplementary material
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Yes, each tables and ﬁgures contain such information on
the distributions when relevant.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Our code mostly
depends on the POT and Pytorch librairies which are quoted in section 2.
(b) Did you mention the license of the assets?
[Yes] Those assets have been following the
format prescribed in their respective websites/papers.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Some implementations of the methods reproduced in the benchmark section 3.2 depend
on assets which were not used initially for our own method.
Therefore we quoted them
in the supplementary
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Used datasets are public ones, we quoted their designers as sign
of consent.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No] Used datasets are well-known public ones that
do not contain such content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]