I SO NN: I SOMORPHIC N EURAL N ETWORK FOR G RAPH R EPRESENTATION L EARNING AND C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning models have achieved huge success in numerous ﬁelds, such as computer vision and natural language processing.
However, unlike such ﬁelds, it is hard to apply traditional deep learning models on the graph data due to the ‘node-orderless’ property.
Normally, adjacency matrices will cast an artiﬁcial and random node-order on the graphs, which renders the performance of deep models on graph classiﬁcation tasks extremely erratic, and the representations learned by such models lack clear interpretability.
To eliminate the unnecessary nodeorder constraint, we propose a novel model named Iso morphic N eural N etwork (I SO NN), which learns the graph representation by extracting its isomorphic features via the graph matching between input graph and templates.
I SO NN has two main components: graph isomorphic feature extraction component and classiﬁcation component.
The graph isomorphic feature extraction component utilizes a set of subgraph templates as the kernel variables to learn the possible subgraph patterns existing in the input graph and then computes the isomorphic features.
A set of permutation matrices is used in the component to break the node-order brought by the matrix representation.
Three fully-connected layers are used as the classiﬁcation component in I SO NN.
Extensive experiments are conducted on benchmark datasets, the experimental results can demonstrate the effectiveness of
I SO NN, especially compared with both classic and state-of-the-art graph classiﬁcation methods.
I NTRODUCTION
The graph structure is attracting increasing interests because of its great representation power on various types of data.
Researchers have done many analyses based on different types of graphs, such as social networks, brain networks and biological networks.
In this paper, we will focus on the binary graph classiﬁcation problem, which has extensive applications in the real world.
For example, one may wish to identify the social community categories according to the users’ social interactions (Gao et al., 2017), distinguish the brain states of patients via their brain networks (Wang et al., 2017), and classify the functions of proteins in a biological interaction network (Hamilton et al., 2017).
To address the graph classiﬁcation task, many approaches have been proposed.
One way to estimate the usefulness of subgraph features is feature evaluation criteria based on both labeled and unlabeled graphs (Kong & Yu, 2010).
Some other works also proposed to design a pattern exploration approach based on pattern co-occurrence and build the classiﬁcation model (Jin et al., 2009) or develop a boosting algorithm (Wu et al., 2014).
However, such works based on BFS or DFS cannot avoid computing a large quantity of possible subgraphs, which causes high computational complexity though the explicit subgraphs are maintained.
Recently, deep learning models are also widely used to solve the graph-oriented problems.
Although some deep models like MPNN (Gilmer et al., 2017) and GCN (Kipf & Welling, 2016) learn implicit structural features, the explict structural information cannot be maintained for further research.
Besides, most existing works on graph classiﬁcation use the aggregation of the node features in graphs as the graph representation (Xu et al., 2018; Hamilton et al., 2017), but simply doing aggregation on the whole graph cannot capture the substructure precisely.
While there are other models can capture the subgraphs, they often needs more complex computation and mechanism (Wang et al., 2017; Narayanan et al., 2017).
However, we should notice that when we deal with the graph-structured data, different node-orders will result in very different adjacency matrix representations for most existing deep models which take the adjacency matrices as input if there is no other information on graph.
Therefore, compared with the original graph, matrix naturally poses a redundant constraint on the graph node-order.
Such a node-order is usually unnecessary and manually deﬁned.
The different graph matrix representations brought by the node-order differences may render the learning performance of the existing models to be extremely erratic and not robust.
Formally, we summarize the encountered challenges in the graph classiﬁcation problem as follows:
• Explicit useful subgraph extraction.
The existing works have proposed many discriminative models to discover useful subgraphs for graph classiﬁcation, and most of them require manual efforts.
Nevertheless, how to select the contributing subgraphs automatically without any additional manual involvement is a challenging problem.
• Graph representation learning.
Representing graphs in the vector space is an important task since it facilitates the storage, parallelism and the usage of machine learning models for the graph data.
Extensive works have been done on node representations (Grover & Leskovec, 2016; Lin et al., 2015; Lai et al., 2017; Hamilton et al., 2017), whereas learning the representation of the whole graph with clear interpretability is still an open problem requiring more explorations.
• Node-order elimination for subgraphs.
Nodes in subgraphs are orderless, whereas the matrix representations of subgraphs cast an unnecessary order on the nodes, which also renders the features extracted with the existing learning models, e
g., CNN, to be useless for the graphs.
Thus, how to break such a node-order constraint for subgraphs is challenging.
• Efﬁcient matching for large subgraphs.
To break the node-order, we will try all possible node permutations to ﬁnd the best permutation for a subgraph.
Clearly, trying all possible permutaions is a combinatorial explosion problem, which is extremly time-comsuming for ﬁnding large subgraph templates.
The problem shows that how to accelerate the proposed model for large subgraphs also needs to be solved.
In this paper, we propose a novel model, namely Iso morphic N eural N etwork (I SO NN) and its variants, to address the aforementioned challenges in the graph representation learning and classiﬁcation problem.
I SO NN is composed of two components: the graph isomorphic feature extraction component and the classiﬁcation component, aiming at learning isomorphic features and classifying graph instances, respectively.
In the graph isomorphic feature extraction component, I SO NN automatically learns a group of subgraph templates of useful patterns from the input graph.
I SO NN makes use of a set of permutation matrices, which act as the node isomorphism mappings between the templates and the input graph.
With the potential isomorphic features learned by all the permutation matrices and the templates, I SO NN adopts one min-pooling layer to ﬁnd the best node permutation for each template and one softmax layer to normalize and fuse all subgraph features learned by different kernels, respectively.
Such features learned by different kernels will be fused together and fed as the input for the classiﬁcation component.
I SO NN further adopts three fully-connected layers as the classiﬁcation component to project the graph instances to their labels.
Moreover, to accelerate the proposed model when dealing with large subgraphs, we also propose two variants of I SO NN to gurantee the efﬁciency.
R ELATED W ORK
Our work relates to subgraph mining, graph neural networks, network embedding as well as graph classiﬁcation.
We will discuss them brieﬂy in the followings.
Subgraph Mining and Graph Kernel Methods: Mining subgraph features from graph data has been studied for many years.
The aim is to extract useful subgraph features from a set of graphs by adopting some speciﬁc criteria.
One classic unsupervised method (i.e., without label information) is gSpan (Yan & Han, 2002), which builds a lexicographic order among graphs and map each graph to a unique minimum DFS code as its canonical label.
GRAMI (Elseidy et al., 2014) only stores templates of frequent subgraphs and treat the frequency evaluation as a constraint satisfaction problem to ﬁnd the minimal set.
For the supervised model (i.e., with label information), CORK utilizes labels to guide the feature selection, where the features are generated by gSpan (Thoma et al., 2009).
Due to the mature development of the sub-graph mining ﬁeld, subgraph mining methods have also been adopted in life sciences (Mrzic et al., 2018).
Moreover, several parallel computing based methods (Qiao et al., 2018; Hill et al., 2012; Lin et al., 2014) have proposed to reduce the time cost.
On the other hand, graph kernel methods are also applied to discover the subgraph structures (Kashima et al., 2003; Vishwanathan et al., 2010; Ga¨uzere et al., 2012; Shervashidze et al., 2011).
Among them, most existing works focus on the graph with node labels and the kernels methods only computes the similarity between pairwise graphs.
Yet, in this paper, we are handling the graph without node labels.
Moreover, we can not only compute the similarity between pairwise graphs, but also learn subgraph templates, which can be further analyzed.
Graph Neural Network and Network Embedding: Graph Neural Networks (Monti et al., 2017;
Atwood & Towsley, 2016; Masci et al., 2015; Kipf & Welling, 2016; Battaglia et al., 2018) have been studied in recent years because of the prosperity of deep learning.
Traditional deep models cannot be directly applied to graphs due to the special data structure.
The general graph neural model MoNet (Monti et al., 2017) employs CNN architectures on non-Euclidean domains such as graphs and manifold.
The GCN proposed in (Kipf & Welling, 2016) utilizes the normalized adjacency matrix to learn the node features for node classiﬁcation; (Bai et al., 2018) proposes the multiscale convolutional model for pairwise graph similarity with a set matching based graph similarity computation.
However, these existing works based on graph neural networks all fail to investigate the node-orderless property of the graph data and to maintain the explicit structural information.
Another important topic related to this paper is network embedding (Bordes et al., 2013; Lin et al., 2015; Lai et al., 2017; Abu-El-Haija et al., 2018; Hamilton et al., 2017), which aims at learning the feature representation of each individual node in a network based on either the network structure or attribute information.
Distinct from these network embedding works, the graph representation learning problem studied in this paper treats each graph as an individual instance and focuses on learning the representation of the whole graph instead.
Graph Classiﬁcation: Graph classiﬁcation is an important problem with many practical applications.
Data like social networks, chemical compounds, brain networks can be represented as graphs naturally and they can have applications such as community detection (Zhang et al., 2018), anti-cancer activity identiﬁcation (Kong et al., 2013; Kong & Yu, 2010) and Alzheimer’s
patients diagnosis (Tong et al., 2017; 2015) respectively.
Traditionally, researchers mine the subgraphs by
DFS or BFS (Saigo et al., 2009; Kong et al., 2013), and use them as the features.
With the rapid development of deep learning (DL), many works are done based on DL methods.
GAM builds the model by RNN with self-attention mechanism (Lee et al., 2018).
DCNN extend CNN to general graph-structured data by introducing a ‘diffusion-convolution’ operation (Atwood & Towsley, 2016).
T ERMINOLOGY AND P ROBLEM D EFINITION
In this section, we will deﬁne the notations and the terminologies used in this paper and give the formulation for the graph classiﬁcation problem.
3. N OTATIONS
In the following sections, we will use lower case letters like x to denote scalars, lower case bold letters (e.g. x ) to represent vectors, bold-face capital letters (e.g. X ) to show the matrices.
For tensors or sets, capital calligraphic letters are used to denote them.
We use x i to represent the i -th element in x .
Given a matrix X , we use X ( i, j ) to express the element in i -th row and j -th column.
For i -th row vector and j -th column vector, we use X ( i, :) and X (: , j ) to denote respectively.
Moreover, notations x (cid:62) and X (cid:62) denote the transpose of vector x and matrix X respectively.
Besides, the F -norm of matrix X can be represented as (cid:107) X (cid:107) F = ( (cid:80) i,j | X i,j | 2 ) 12 .
3. P ROBLEM F ORMULATION
Many real-world inter-connected data can be formally represented as the graph-structured data.
[Equation]
: Formally, a graph can be represented as G =
( V , E ) , where the sets V and E denote the nodes and links involved in the graph, respectively.
Some representative examples include the human brain graphs (where the nodes denote brain regions and links represent the correlations among these regions), biological molecule graphs (with the nodes represent the atoms and links denote the atomic bonds), as well as the geographical graphs in the ofﬂine world (where the nodes denote the communities and the links represent the commute routes among communities).
Meanwhile, many concrete real-world application problems, e.g., brain graph based patient disease diagnosis, molecule function classiﬁcation and community vibrancy prediction can also be formulated as the graph classiﬁcation problems.
[Table]
Graph Isomorphic Layer Min-pooling Layer
Softmax Layer
[Table] [Table]
+B8/gBS3Y0l</latexit> <latexit sha1_base64="/Vt+0RdSS5A/sbViRuabh3HoiBU=">AAAB6nicbVA9TwJBEJ3DL8Qv1NJmI5hYkTsaLYkWWmIUJIEL2VvmYMPe3mV3z4Rc+Ak2Fhpj6y+y89+4wBUKvmSSl/dmMjMvSATXxnW/ncLa+sbmVnG7tLO7t39QPjxq6zhVDFssFrHqBFSj4BJbhhuBnUQhjQKBj8H4euY/PqHSPJYPZpKgH9Gh5CFn1FjpvnpT7Zcrbs2dg6wSLycVyNHsl796g5ilEUrDBNW667mJ8TOqDGcCp6VeqjGhbEyH2LVU0gi1n81PnZIzqwxIGCtb0pC5+nsio5HWkyiwnRE1I73szcT/vG5qwks/4zJJDUq2WBSmgpiYzP4mA66QGTGxhDLF7a2EjaiizNh0SjYEb/nlVdKu1zy35t3VK42rPI4inMApnIMHF9CAW2hCCxgM4Rle4c0Rzovz7nwsWgtOPnMMf+B8/gBS3Y0l</latexit>
Problem Deﬁnition : Formally, given a graph set G = { G 1 , G 2 , · · · , G n } with a small number of labeled graph instances, the graph classiﬁcation problem aims at learning a mapping, i.e., f : G →
Y , to project each graph instance into a pre-deﬁned label space Y = { +1 , − 1 } .
In this paper, we will take the graph binary classiﬁcation as an example to illustrate the problem setting for I SO NN.
A simple extension of the model can be applied to handle more complicated learning scenarios with multi-class or multi-label as well.
P ROPOSED M ETHOD
The overall architecture of I SO NN is shown in Figure 1.
The I SO NN framework includes two main components: graph isomorphic feature extraction component and classiﬁcation component.
The graph isomorphic feature extraction component includes a graph isomorphic layer, a minpooling layer as well as a softmax layer and the classiﬁcation component is composed by three fully-connected layers.
They will be discussed in detail in the following subsections.
4. G RAPH I SOMORPHIC F EATURE E XTRACTION C OMPONENT
Graph isomorphic feature extraction component targets at learning the graph features.
To achieve that objective, I SO NN adopts an automatic feature extraction strategy for graph representation learning.
In I SO NN, one graph isomorphic feature extraction component involves three layers: the graph isomorphic layer, the min-pooling layer and the softmax layer.
In addition, we can further construct a deep graph isomorphic neural network by applying multiple isomorphic feature extraction components on top of each other, i.e. , apply the combination of ”graph isomorphic layer, min pooling layer, softmax layer” several times.
For the second and latter components, they will be used on every feature matrix learned by the combination of channels of all former components.
4.1. G RAPH I SOMORPHIC L AYER
Graph isomorphic layer is the ﬁrst effective layer in deep learning that handles the node-order restriction in graph representations.
Assume we have a graph G =
{V , E} , and its adjacency matrix to be A ∈ R |V|×|V| .
In order to ﬁnd the existence of speciﬁc subgraph patterns in the input graph,
I SO NN matches the input graph with a set of subgraph templates.
Each template is denoted as a kernel variable K i ∈
R k × k , ∀ i ∈ { 1 , 2 , · · · , c } .
Here, k denotes the node number in subgraphs and c is the channel number (i.e., total template count).
Meanwhile, to match one template with regions in the input graph (i.e., sub-matrices in A ), we use a set of permutation matrices, which map both rows and columns of the kernel matrix to the subgraphs effectively.
The permutation matrix can be represented as P ∈ { 0 , 1 } k × k that shares the same dimension with the kernel matrix.
Therefore, given a kernel matrix K i and a sub-matrix M ( s,t ) ∈ R k × k in A (i.e., a region in the input graph G and s, t ∈ { 1 , 2 , · · · , ( |V| − k + 1) } denotes a starting index pair in A ), there may exist k ! different such permutation matrices.
The optimal should be the matrix P ∗ that minimizes the following term.
[Equation]
where P =
{ P 1 , P 2 , · · · , P k ! } covers all the potential permutation matrices.
Formally, the isomorphic feature extracted based on the kernel K i for the regional sub-matrix M ( s,t ) in A can be represented as
[Equation]
where vector ¯ z i, ( s,t ) ∈
R k ! contains entry
¯ z i, ( s,t
[Equation]
{ 1 , 2 , · · · , k ! } denoting the isomorphic features computed by the j -th permutation matrix P j ∈ P .
As indicated by the Figure 1, I SO NN computes the ﬁnal isomorphic features for the kernel K i via two steps: (1) computing all the potential isomorphic features via different permutation matrices with the graph isomorphic layer, and (2) identifying and fusing the optimal features with the minpooling layer and softmax layer to be introduced as follows.
By shifting one kernel matrix K i on regional sub-matrices, I SO NN extracts the isomorphic features on the matrix A , which can be denoted as a 3-way tensor ¯ Z i ∈ R k !
× ( |V|− k +1) ×
( |V|− k +1) , where ¯ Z i (1 : k ! , s, t ) =
¯
z i, ( s,t ) (1 : k !) .
In a similar way, we can also compute the isomorphic feature tensors based on the other kernels, which can be denoted as ¯ Z 1 , ¯ Z 2 , · · · , ¯ Z c respectively.
4.1. M IN - POOLING L AYER
Given the tensor ¯
Z i computed by K i in the graph isomorphic layer, I SO NN will identify the optimal permutation matrices via the min-pooling layer.
Formally, we can represent results of the optimal permutation selection with ¯ Z i as matrix Z i :
[Equation]
The min-pooling layer learns the optimal matrix Z i for kernel K i along the ﬁrst dimension (i.e., the dimension indexed by different permutation matrices), which can effectively identify the isomorphic features created by the optimal permutation matrices.
For the remaining kernel matrices, we can also achieve their corresponding graph isomorphic feature matrices as Z 1 , Z 2 , · · · , Z c respectively.
4.1. S OFTMAX L AYER
Based on the above descriptions, a perfect matching between the subgraph templates with the input graph will lead to a very small isomorphic feature, e.g., a value approaching to 0 .
If we feed the small features into the classiﬁcation component, the useful information will vanish and the relative useless information ( i.e. , features learned by the subgraphs dismatch the kernels) dominates the learning feature vector in the end.
Meanwhile, the feature values computed in Equation (3) can also be in different scales for different kernels.
To effectively normalize these features, we propose to apply the softmax function to matrices Z 1 , Z 2 , · · · , Z c across all c kernels.
Compared with the raw features, e.g., Z i , softmax as a non-linear mapping can also effectively highlight the useful features in Z i by rescaling them to relatively larger values especially compared with the useless ones.
Formally, we can represent the fused graph isomorphic features after rescaling by all the kernels as a 3-way tensor
Q , where slices along ﬁrst dimension can be denoted as:
[Equation]
4. C LASSIFICATION C OMPONENT
After the isomorphic feature tensor Q is obtained, we feed it into a classiﬁcation component.
Let q denote the ﬂattened vector representation of feature tensor Q , and we pass it to three fully-connected layers to get the predicted label vector ˆ y .
For the graph binary classiﬁcation, suppose we have the ground truth y
=
( y g 1 , y g 2 ) and the predicted label vector ˆ y g
=
(ˆ y g 1 , ˆ y g 2 ) for the sample g from the training batch set B .
We use cross-entropy as the loss function in I SO NN.
Formally, the fullyconnected (FC) layers and the objective function can be represented as follows respectively:
FC Layers:
[Equation]
where W i and b i represent the weights and biases in i -th layer respectively for i ∈ { 1 , 2 , 3 } .
The σ denotes the adopted the relu activation function.
To train the proposed model, we adopt the back propagation algorithm to learn both the subgraph templates and the other involved variables.
4. M ORE D ISCUSSIONS ON G RAPH I SOMORPHIC F EATURE E XTRACTION IN I SO NN
Before introducing the empirical experiments to test the effectiveness of I SO NN, we would like to provide more discussions about the computation time complexity of the graph isomorphic feature extraction component involved in I SO NN.
Formally, given the input graph G with n = |V| nodes, by shifting the kernel variables (of size k × k ) among the dimensions of the corresponding graph adjacency matrix, we will be able to obtain ( n − k +1) 2 regional sub-matrices (or O ( n 2 ) regional submatrices for notation simplicity).
Here, we assume I SO NN has only one isomorphic layer involving c different kernels.
In the forward propagation, the introduced time cost in computing the graph isomorphic features can be denoted as O ( ck ! k 3 n 2 ) , where term k ! is introduced in enumerating all the potential permutation matrices and k 3 corresponds to the matrix multiplication time cost.
According to the notation, we observe that n is ﬁxed for the input graph.
Once the kernel channel number c is decided, the time cost notation will be mainly dominated by k .
To lower down the above time complexity notation, in this part, we propose to further improve I SO NN from two perspectives: (1) compute the optimal permutation matrix in a faster manner, and (2) use deeper model architectures with small-sized kernels.
4.3. F AST P ERMUTATION M ATRIX C OMPUTATION
Instead of enumerating all the permutation matrices in the graph isomorphic feature extraction as indicated by Equations (2)-(3), here we introduce a fast way to compute the optimal permutation matrix for the provided kernel variable matrix, e.g., K i , and input regional sub-matrix, M ( s,t ) , directly according to the following theorem.
T HEOREM 1 Formally, let the kernel variable K i and the input regional sub-matrix M ( s,t ) be k × k real symmetric matrices with k distinct eigenvalues α 1 > α 2 > · · · > α k and β 1 > β 2 > · · · > β k , respectively, and their eigendecomposition be represented by
[Equation]
where U K i and U M ( s,t ) are orthogonal matrices of eigenvectors and Λ K i =
diag ( α j ) , Λ M ( s,t ) =
diag ( β j ) .
The minimum of || PK i P (cid:62) − M ( s,t ) || 2 is attained for the following P ’s:
[Equation]
where S ∈ S =
{ diag ( s 1 , s 2 , · · · , s k ) | s i = 1 or − 1 } .
The proof of the theorem will be provided in appendix.
In computing the optimal permutation matrix
P ∗ , trials of different S will be needed.
Meanwhile, to avoid such time costs, we introduce to take the upper bound value of U M ( s,t ) SU (cid:62) K i as the approximated optimal permutation matrix instead,
which together with the corresponding optimal feature z i, ( s,t ) can be denoted as follows:
[Equation]
where |·| denotes the absolute value operator and | U M ( s,t )
||
U (cid:62) K i | ≥ U M ( s,t ) SU (cid:62) K
i hold for ∀ S ∈ S .
By replacing Equations 2-3 with Equation 7, we can compute the optimal graph isomorphic feature for the kernel K i and input regional sub-matrix M ( s,t ) with a much lower time cost.
Furthermore, since the eigendecomposition time complexity of a k × k matrix is O ( k 3 ) , based on the above theorem, we will be able to lower down the total time cost in graph isomorphic feature extraction to
O ( ck 3 n 2 ) , which can be further optimized with the method introduced in the following subsection.
4.3. D EEP G RAPH I SOMORPHIC F EATURE E XTRACTION
Here, we will illustrate the advantages of deep I SO NN model with small-sized kernels compared against shallow I SO NN model with large kernels.
Since graph isomorphic layer is the main functional layer to learn subgraph features, we simply use multi-layer for short to denote the mutiple graph isomorphic feature extraction component.
In Figure 2, we provide an example two I SO NN models with different model architectures
• the left model has one single layer and 6 kernels, where the kernel size k = 12 ; • the right model has two layers: layer 1 involves 2 kernels of size 3 , and layer 2 involveskernels of size 4 .
[Figure]
By comparing these two different models, we observe that they have identical representation learning capacity.
However, the time cost in feature extraction introduced by the left model is much higher than that introduced by the right model, which can be denoted as O (6(12 3 ) n 2 ) and O (2(3 3 ) n 2 + 3(4 3 ) n 2 ) , respectively.
Therefore, for the I SO NN model, we tend to use small-sized kernels.
Formally, according to the fast method provided in the previous part, given a 1-layer I SO NN model with c large kernels of size k , its graph isomorphic feature extraction time complexity can be denoted as O ( ck 3 n 2 ) .
Inspired by Figure 2, without affecting the representation capacity, such a model can be replaced by a max {(cid:100) log c 2 (cid:101) , (cid:108) log k 3 (cid:109) } -layers deep I SO NN model instead, where each layer involves 2 kernels of size 3 .
The graph isomorphic feature extraction time complexity of the deep model will be
O (cid:16) (max {(cid:100) log c 2 (cid:101) , (cid:108) log k 3 (cid:109) } ) · 2 · 3 3 n 2 (cid:17) (or O (cid:16) (max {(cid:100) log c (cid:101) , (cid:108) log k (cid:109) } ) · n 2 (cid:17) for simplicity).
E XPERIMENTS
To evaluate the performance of I SO NN, in this section, we will talk about the experimental settings as well as ﬁve benchmark datasets.
Finally, we will discuss the experimental results with parameter analyses on kernel size , channel number and time complexity.
5. E XPERIMENTAL S ETTINGS
In this subsection, we will use ﬁve real-world benchmark datasets: HIV-fMRI (Cao et al., 2015a),
HIV-DTI (Cao et al., 2015a), BP-fMRI
(Cao et al., 2015b), MUTAG 1 and PTC 1 .
Both HIV-fMRI and HIV-DTI have 56 positive instances and 21 negative instances.
Also, graph instances in both of them are represented as 90 × 90 matrices (Cao et al., 2015a).
BP-fMRI has 52 positive andnegative instances and each instance is presented by an 82 × 82 matrix Cao et al.
(2015b).
MUTAG and PTC are two datasets which have been widely used in academia Xu et al.
; Shervashidze et al.
MUTAG has 125 positive and 63 negative graph instances with graph size 28 × 28 .
PTC is a relative large dataset, which has 152 positive and 192 negative graph instances with graph size 109 × 109 .
With these datasets, we ﬁrst introduce the comparison methods used in this paper and then talk about the experimental setups and the adopted evaluation metrics in detail.
5.1. C OMPARISON M ETHODS
• I SO NN & I SO NN-fast : The proposed method I SO NN uses a set of template variables as well as the permutation matrices to extract the isomorphic features and feed these features to the classiﬁcation component.
The variant model named I SO NN-fast uses the Equationto compute the optimal permutation matrices and other settings remain unchanged.
• Freq : The method uses the top- k frequent subgraphs as its features.
This is also an unsupervised feature selection method based on frequency.
• AE : We use the autoencoder model (AE) (Vincent et al., 2010) to get the features of graphs without label information.
It is an unsupervised learning method, which learns the latent representations of connections in the graphs without considering the structural information.
• CNN: It is the convolutional model (Krizhevsky et al., 2012) learns the structural information within small regions of the whole graph.
We adopt one convolution layer and
three fully-connected layer to be the classiﬁcation module.
[Table]
• SDBN :
A model proposed in (Wang et al., 2017), which reorders the nodes in the graph ﬁrst and then feeds the reordered graph into an augmented CNN.
In this way, it not only learns the structural information but also tries to minimize the effect of the order constraint.
• WL: WL (Shervashidze et al., 2011) is a classic algorithm to do the graph isomorphism test.
For the graph classiﬁcation, we computes the similarity scores for test graphs and train graph.
The mean of all similarity scores between each test graph and train graphs will be used to do the classiﬁcation.
• GCN: GCN is proposed in (Kipf & Welling, 2016) use the adjacent matrix to learn the implicit structure information in graphs.
Here, we use two graph convolutional layers to learn node features and then take the all nodes features as the graph features.
One fullyconnected layer will be used as graph classiﬁcation module.
• GIN: GIN is proposed in (Xu et al., 2018) can be used to do graph classiﬁcation with node features.
We adopt the same experimental setting as GIN-0 stated in (Xu et al., 2018).
5.1. E XPERIMENTAL S ETUP AND E VALUATION M ETRICS
In our experiments, to make the results more reliable, we partition the datasets into 3 folds and then set the ratio of train/test according to 2 : 1 , where two folds are treated as the training data and the remaining one is the testing data.
We select top-100 features for Freq as stated in (Wang et al., 2017) with a three layer MLP classiﬁer, where the neuron numbers are 1024, 128, 2.
For Auto-encoder, we apply the two-layer encoder and two-layer decoder.
For the CNN, we apply the one convolutional layer with the size 5 × 5 × 50 , a max-pooling layer with kernel size 2 × 2 , one gating relu layer as activation layer and we set parameters in the classiﬁcation module the same as Freq classiﬁer.
For the SDBN, we set the architecture as follows: we use two layers of ”convolution layer + max pooling layer + activation layer ” and concatenate a fully connected layer with 100 neurons as well as an activation layer, where the parameters are the same as those in CNN.
We also set the dropout rate in SDBN being 0.5 to avoid overﬁtting.
For WL kernel, if the average similarity score for one test graph greater than 0.5, we assign the test graph positive label, otherwise, assign negative label.
We follow the setting in (Kipf & Welling, 2016) and (Xu et al., 2018) to do GCN and GIN-0.
Here, to make a fair comparison, we will use the adjacency matrices as features ( i.e. , no node label information) for WL, GCN and GIN.
In the experiments, we set the kernel size k in the isomorphic layer for three datasets as 2, 4, 3, 4, 4 respectively, and then set the parameters in classiﬁcation component the same as those in Freq classiﬁer.
In this experiment, we adopt Adam optimizer and the set the learning rate η = 0 .
001 , and then we report the average results on balanced datasets.
5. E XPERIMENTAL R ESULTS
In this section, we investigate the effectiveness of the learned subgraph-based graph feature representations for graphs.
We adopt one isomorphic layer where the kernel size k = 2 and channel number c = 3 for HIV-fMRI, one isomorphic layer with ( k = 4 , c = 2) , ( k = 3 , c = 1) , ( k = 4 , c = 1) and ( k = 4 , c = 2) for the HIV-DTI, BP-fMRI, MUTAG and PTC, respectively.
The results are shown in Table 1.
From that table, we can observe that I SO NN outperforms all other baseline methods on these all datasets.
We need to remark that I SO NN and I SO NN-fast are very close on MUTAG, and I SO NN has the best performance in total on PTC.
Compared with Freq, the proposed method achieves a better performance without searching for all possible subgraphs manually.
AE has almost
5 k 0.
AccuracyF (a) HIV-fMRI   5 k 0.
AccuracyF (b) HIV-DTI   5 k 0.
AccuracyF (c) BP-fMRI   5 k 0.
AccuracyF (d) MUTAG   5 k 0.
AccuracyF (e) PTC the worst performance among all comparison methods.
This is because the features learned from
AE do not contain any structural information.
For HIV-DTI, AE gets 0 in F1.
This is because the dataset contains too many zeros, which makes the AE learns trivial features.
Also, for PTC, its F is higher than other models, but the accuracy only get 50.0, which indicates AE actually have a bad performance since it cannot discriminate the classes of the instances ( i.e. , predicting all positive classes).
CNN performs better than AE but still worse than I SO NN.
The reason can be that it learns some structural information but fails to learn representative structural patterns.
SDBN is designed for brain images, so it may not work for MUTAG and PTC.
One possible reason for WL got bad results is the isomorphism test is done on the whole graph, which may lead to erratic results.
GCN performs better than GIN but worse than I SO NN, showing that GCN can learn some sturctual information without node labels, but GIN cannot work with the adjacency matrix as input.
I SO NN-fast achieves the best scores on MUTAG and second-best on HIV-fMRI, yet worse than several other methods on other datasets.
This can be the approximation on P may impair the performance.
Comparing I SO NN with AE, I SO NN achieves better results.
This means the structural information is more important than only connectivity information for the classiﬁcation problem.
If compared with
CNN, the results also show the contribution of breaking the node-order in learning the subgraph templates.
Similar to SDBN, I SO NN also ﬁnds the features from subgraphs, but I SO NN gets better performance with more concise architecture.
Contrasting with GCN and GIN, I SO NN can maintain the explict subgraph structures in graph representations, while the GCN and GIN simply use the aggragation of the neighboring node features, losing the graph-level substructure infomation.
5. P ARAMETER A NALYSIS
To further study the proposed method, we will discuss the effects of different kernel size and channel number in I SO NN.
The model convergence analysis will be provided in appendix.
• Kernel Size : We show the effectiveness of different k in Figure 3.
Based on the previous statement, parameter k can affect the ﬁnal results since it controls the size of learned subgraph templates.
To investigate the best kernel size for each dataset, we ﬁx the channel number c = 1 .
As Figure 3 shows, different datasets have different appropriate kernel sizes.
The best kernel sizes are 2, 4, 3, 4, 4 for the three datasets respectively.
• Channel Number :
We also study the effectiveness of multiple channels (i
e., multiple templates in one layer).
To discuss how the channel number inﬂuences the results, we choose the best kernel size for each dataset (i.e., 2, 4, 3, 4, 4 respectively).
From all subﬁgures in Figure 4, we can see that the differences among the different channel numbers by using only one isomorphic layer.
As shown in Figure 4, I SO NN achieves the best results by c = 3 , 2 , 1 , 1 , 2 , respectively, which means the increase of the channel number can improve the performance, but more channels do not
necessarily lead to better results.
The reason could be the more templates we use, the more complex our model would be.
With such a complex model, it is easy to learn an overﬁtting model on train data, especially when the dataset is quite small.
Thus, increasing the channel number can improve the performance but the effectiveness will still depend on the quality and the quantity of the dataset.
[Figure] [Figure]
4 c
[Figure]
AccuracyF
[Figure]
HIV
[Figure]
DTI   4 c 0.
AccuracyF
[Figure] [Figure]
[Figure]
5. T IME C OMPLEXITY S TUDY
To study the efﬁciency of I SO NN and I SO NN-fast, we collect the actual running time on training model, which is shown in Figure 5.
In both Figures 5(a) and 5(b) 2 , the x-axis denotes its value for k or c and the y-axis denotes the time cost with different parameters.
From Figure 5(a), four lines show the same pattern.
When the k increases, the time cost grows exponentially.
This pattern can be directly explained by the size of the permutation matrix set.
When we increase the kernel size by one, the number of corresponding permutation matrices grows exponentially.
While changing c , shown in Figure 5(b), it is easy to observe that those curves are basically linear with different slopes.
This is also natural since whenever we add one channel, we only need to add a constant number of the permutation matrices.
To study the efﬁciency of I SO NN-fast, Figure 5(c) shows the running times of I
SO NN
and I SO NN-fast on MUTAG.
As it shows, I SO NN-fast use less time when the kernel size greater than 4 , otherwise I
SO NN
and I SO NN will have little difference since the eigen decomposition has nearly the same time complexity as calculating all possible node permutaions.
The results also verify the theoretical time complexity analysis in 4.3.
C ONCLUSION
In this paper, we proposed a novel graph neural network named I SO NN to solve the graph classiﬁcation problem.
I SO NN consists of two components: (1) isomorphic component, where a set of permutation matrices is used to break the randomness order posed by matrix representation for a bunch of templates and one min-pooling layer and one softmax layer are used to get the best isomorphic features, and (2) classiﬁcation component, which contains three fully-connected layers.
We further discuss the two efﬁcient variants of I SO NN to accelerate the model.
Next, we perform the experiments on ﬁve real-world datasets.
The experimental results show the proposed method outperforms all comparison methods, which demonstrates the superiority of our proposed method.
The experimental analysis on time complexity illustrates the efﬁciency of the I SO NN-fast.
A PPENDIX 7. P ROOF OF T HEOREM
Before giving the proof of Theorem 1, we need to introduce Lemma 1 ﬁrst.
L EMMA
If A and B are Hermitian matrices with eigenvalues α 1 ≥ α 2 ≥ · · · ≥ α n and β 1 ≥ β 2 ≥ · · · ≥ β n respectively, then ||
A − B || ≥ (cid:80) ni =
( α i − β i ) 2 .
Based on Lemma 1, we can derive the proof of Theorem 1 as follows.
P ROOF 1 From Lemma 1, Equation 9 holds for any orthogonal matrix R since the eigenvalues of
RK i R (cid:62) are the same as those of K i .
[Equation]
On the other hand, if we use P in 7, we have
[Equation]
where we use the equations that || UX
|| =
|| UX (cid:62)
||
= ||
X || for any orthogonal matrix U and
SΛ K i S
=
S
2 Λ K i
= Λ K
i since S and Λ K i are both orthogonal matrices and S
= I .
Moreover, it is clear that
[Equation]
because of the elements in S are either − 1 or +1 .
Also, since each row vector of U M ( s,t ) and U K i is unit vector, thus we can have
[Equation]
If there exists a perfect permutaion matrix, P ∗ , then there exists such S ∗ , s.t.
[Equation]
Thus, based on Equation (11), Equation (12) and Equation (13), we can get
[Equation]
This means that P maximizes tr ( P (cid:62) | U M ( s,t )
||
U (cid:62) K i | ) since tr ( P (cid:62) | U M ( s,t ) ||
U (cid:62) K i | ) for any permuta- tion matrix P. Therefore, when K i and M ( s,t ) are isomorphic, the optimal permutation matrix can be obtained as a permutation matrix P which maximizes tr ( P (cid:62) | U M ( s,t )
||
U (cid:62) K i | ) .
Therefore, we take
P ∗ =
| U M ( s,t )
||
U (cid:62) K i | directly.
7. C ONVERGENCE A NALYSIS
The Figure 6 shows the convergence trend of I SO NN on ﬁve datasets, where the x-axis denotes the epoch number and the y-axis is the training loss, respectively.
From these sub-ﬁgures, we can know that the proposed method can achieve a stable optimal solution within 50 iterations except for MUTAG (it needs almost 130 epochs to converge), which also illustrates our method would converge relatively fast.
[Figure]