C APSULE G RAPH N EURAL N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance.
However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representation may not sufﬁce to preserve the node/graph properties efﬁciently, resulting in sub-optimal graph embeddings.
Inspired by the Capsule Neural Network (CapsNet) (Sabour et al., 2017), we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms.
By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level.
As a result, our model generates multiple embeddings for each graph to capture graph properties from different aspects.
The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs.
Our extensive evaluations with 10 graph-structured datasets demonstrate that
CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven.
It outperforms other SOTA techniques on several graph classiﬁcation tasks, by virtue of the new instrument.
I NTRODUCTION
GNNs is a general type of deep-learning architectures that can be directly applied to structured data.
These architectures are mainly generalized from other well-established deep-learning models like
CNN (Krizhevsky et al., 2012) and RNN (Mikolov et al., 2010).
In this paper, we mainly focus on
Convolution-based Graph Neural Networks which attract increasing interest recently.
Convolution operation can be embedded into Graph Neural Networks from spectral or spatial perspective.
Bruna et al.
(2013) deﬁnes the convolution of the graph in the Fourier domain which needs to calculate the eigendecomposition of the graph Laplacian.
This method is computationally expensive and the ﬁlters they deﬁned are non-spatially localized.
Later, Henaff et al.
(2015) introduces Chebyshev expansion of the graph Laplacian to avoid computing eigenvectors and Kipf & Welling (2016) proposes to do convolution within 1-step neighbor nodes to reduce the complexity.
From the spatial perspective, Hamilton et al.
(2017) and Zhang et al.
(2018) propose to deﬁne a node receptive-ﬁeld and do convolution within this ﬁeld during which the information of each node as well as their neighbor nodes is gathered and new representation of each node is generated through an activation function.
Both of these two perspectives perform well in node representation learning and a number of variants (Velickovic et al., 2017) are developed based on this convolution idea and some of them have proven to achieve SOTA in various tasks.
The success of GNN in node representation learning has inspired many deep-learning-based approaches to leverage on node embeddings extracted from GNN to generate graph embeddings for graph-based applications.
However, during this procedure, the learned distributed representation of each node will be considered as multiple individual scalar features instead of one vector.
For example, Zhang et al.
(2018) applies element-wise max-pooling to nodes embeddings when generating graph embeddings, Verma & Zhang (2018) generates graph embeddings by computing the elementwise covariance of all nodes.
These operations indicate that the authors capture node features in the form of scalar when they generate graph embeddings which is different from the original intention of GNNs.
To build high-quality graph embeddings, it is important to not only detect the presence of different structures around each node but also preserve their detail properties such as position, direction, connection, etc.
However, encoding these properties information in the form of scalar means activating elements in a vector one-by-one which is exponentially less efﬁcient than encoding them with distributed representations.
This has been identiﬁed discussed in CapsNet.
Inspired by CapsNet, we propose to extend scalar to vector during the procedure of applying GNN to graph representation learning.
Compared with scalar-based neural network, vector-based neural network preserves the information of node/graph properties more efﬁciently.
The technique for extracting features in the form of vectors is proposed in Hinton et al.
and improved in Sabour et al.
(2017) and Hinton et al.
This technique is mainly devised for image processing domain.
In their work, the extracted vector is referred to as capsule (a group of neurons in neural network), so we follow the same notation in our work.
Introducing capsules allows us to use routing mechanism to generate high-level features which we believe is a more efﬁcient way for features encoding.
Compared with max-pooling in CNN in which all information will be dropped except for the most active one, routing preserves all the information from low-level capsules and routes them to the closest high-level capsules.
Besides, this allows to model each graph with multiple embeddings and each embedding reﬂects different properties of the graph.
This is more representative than only one embedding used in other scalar-based approaches.
In this paper, we propose Capsule Graph Neural Network (CapsGNN), a novel deep learning architecture, which is inspired by CapsNet and uses node features extracted from GNN to generate high-quality graph embeddings.
In this architecture, each graph is represented as multiple embeddings and each embedding reﬂects the graph properties from different aspects.
More speciﬁcally, basic node features are extracted in the form of capsules through GNN and routing mechanism is applied to generate high-level graph capsules as well as class capsules.
In the procedure of generating graph capsules, an Attention Module can be applied to tackle graphs in various sizes.
It also assigns different weights to each capsule of each node so that this model focuses on critical parts of the graph.
We validate the performance of generated graph embeddings on classiﬁcation task over 5 biological datasets and 5 social datasets.
CapsGNN achieves SOTA performance on 6 out ofbenchmark datasets and comparable results on the rest.
T-SNE (Maaten & Hinton, 2008) is used to visualize the learned graph embeddings and the results show that different graph capsules indeed capture different information of the graphs.
B ACKGROUND
Here, we provide a brief introduction to GNN, routing mechanism in CapsNet and Attention mechanism which is used in CapsGNN.
2. G RAPH
By deﬁnition, a weighted directed graph can be represented by G =
( V , X , A ) where V = { v 1 , v 2 , ...v N } is the set of nodes and A ∈ { 0 , 1 } N × N is the adjacency matrix.
If there is an edge from v i to v j , then A ij = 1 otherwise A ij = 0 .
X ∈ R N × d represents the features of each node and d is the number of feature channels.
2. G RAPH N EURAL N ETWORK
Graph Convolutional Networks (GCNs) (Kipf & Welling, 2016), a widely used GNN architecture, is one of the key building blocks in our work.
At each layer of the GCN, the convolution operation is applied to each node as well as its neighbors and the new representation of each node is computed through an activation function.
This procedure can be written as:
[Equation]
where Z l ∈ R N × d represents nodes features at the layer l , d represents the number of feature channels and Z
=
X , W l ∈ R d × d (cid:48) is a trainable weights matrix which serves as a channel ﬁlter, f is a nonlinear activation function, T ∈ R N × N is the information transform matrix and it is usually calculated from the adjacency matrix A for guiding the information ﬂowing between nodes.
A complete GNN usually stacks L layers to generate ﬁnal nodes embeddings Z L .
In the architecture proposed by Kipf & Welling (2016), at the l th layer of GNN, the extracted features of each node actually take all its adjacent nodes within l steps into consideration.
So l can be considered as the size of the node receptive-ﬁeld at this layer.
This special property inspired us to use nodes features extracted from different layers to generate the graph capsules.
2. C APSULE N EURAL N ETWORK
The concept of capsules is invented by Hinton’s team (Hinton et al., 2011) and used recently in
Sabour et al.
(2017) and Hinton et al.
CapsNet is designed for image features extraction and it is developed based on CNN.
However, unlike traditional CNN in which the presence of feature is represented with scalar value in feature maps, the features in CapsNet are represented with capsules (vectors).
In Sabour et al.
(2017), the direction of capsules reﬂects the properties of the features and the length of capsules reﬂects the probability of the presence of different features.
The transmission of information between layers follows Dynamic Routing mechanism.
The speciﬁc procedure of
Dynamic Routing can be found in Appendix A for the completeness.
Inspired by CapsNet, the capsule mechanism is adopted and fused with GNN in our proposed CapsGNN to generate graph capsules and class capsules on the basis of node capsules which are extracted from GNN.
Dynamic Routing is applied to update weights between capsules from one layer to the next layer so that the properties captured by node capsules can be propagated to suitable graph capsules.
Thus, each graph is modeled as multiple graph capsules, and then modeled as multiple class capsules.
Different graph capsules reﬂect the properties of the graph from different aspects.
2. A TTENTION M ECHANISM
Attention mechanism is widely applied in image processing (Zheng et al., 2017) and natural language processing domain (Gehring et al., 2016) where it is used to ﬁnd the relevant parts of the input data to the task target.
The main procedure of Attention mechanism is: 1) deﬁning an attention measure which is used to measure the relevance of each part of the input data to the task target.
) normalizing
the
generated attention value.
scaling each part with the normalized attention value
In CapsGNN, we apply Attention mechanism for two purposes: 1) scaling each node capsule so that generated graph capsules from different graphs are comparable even though these graphs are vastly different in sizes.
2) guiding the model to focus on more relevant and important parts of graphs.
C APSULES IN G RAPH N EURAL N ETWORK
In this section, we outline CapsGNN and show how it is used to generate high-quality graph capsules which then can be applied to graph classiﬁcation task.
3. A RCHITECTURE
Figure 1 shows a simpliﬁed version of CapsGNN.
It consists of three key blocks: 1) Basic node capsules extraction block: GNN is applied to extract local vertices features with different receptiveﬁeld and then primary node capsules are built in this block.
2) High level graph capsules extraction block: Attention Module and Dynamic Routing are fused to generate multiple capsules for graphs.
3) Graph classiﬁcation block: Dynamic Routing is applied again to generate ﬁnal graph class capsules for graph classiﬁcation.
The details of each block is explained in the following.
3.1. B ASIC N ODE C APSULES
Firstly, the basic node features are extracted with GNN.
Node degrees are used as node attributes if nodes do not have attributes.
We use the architecture improved by Kipf & Welling (2016) (GCN) as the node features extractor.
The difference is that we extract multi-scale node features from different layers and the extracted features are represented in the form of capsules.
The procedure can be written as:
[Equation]
where W lij ∈ R d × d (cid:48) is the trainable weights matrix.
It serves as the channel ﬁlters from the i th channel at the l th layer to the j th channel at the ( l + 1) th layer.
Here, we choose f ( • ) =
tanh ( • ) as the activation function.
Z l +1 ∈ R N × d (cid:48) , Z 0 = X , ˜
A =
A + I 1 and ˜ D
= (cid:80)
j ˜
A ij .
To preserve features of sub-components with different sizes, we use nodes features extracted from all
GNN layers as well as input layer to generate high-level capsules.
3.1. H IGH -L EVEL G RAPH C APSULES
After getting local node capsules, global routing mechanism is applied to generate graph capsules.
The input of this block contains N sets of node capsules, each set is S n = { s 11 , .., s 1 C 1 , ..., s LC L } , s lc ∈ R d , where C l is the number of channels at the l th layer of GNN, d is the dimension of each capsule.
The output of this block is a set of graph capsules H ∈ R P × d (cid:48) .
Each of the capsules reﬂects the properties of the graph from different aspects.
The length of these capsules reﬂects the probability of the presence of these properties and the angle reﬂects the details of the graph properties.
Before generating graph capsules with node capsules, an Attention Module is introduced to scale node capsules.
Attention Module .
In CapsGNN, primary capsules are extracted based on each node which means the number of primary capsules depends on the size of input graphs.
In this case, if the routing mechanism is directly applied, the value of generated high-level capsules will highly depend on the number of primary capsules (graph size) which is not the ideal case.
Hence, an Attention Module is introduced to combat this issue.
The attention measure we choose is a two-layer fully connected neural network F attn ( • ) .
The number of input units of F attn ( • ) is d × C all where C all = (cid:80) l C l and the number of output units equals to C all .
We apply node-based normalization to generate attention value in each channel and then scale the original node capsules.
The details of Attention Module is shown in Figure 2 and the procedure can be written as:
[Equation]
where ˜ s n ∈ R 1 × C all d is obtained by concatenating all capsules of the node n .
s ( n,i ) ∈
R 1 × d represents the i th capsule of the node n and F attn ( ˜ s n ) ∈ R 1 × C all is the generated attention value.
In 1 I represents identity matrix.
this way, the generated graph capsules can be independent to the size of graphs and the architecture will focus more on important parts of an input graph.
After Attention Module, coordinate addition can be used to preserve the position information of each node during the procedure of generating node capsule votes.
Here, we introduce coordinate addition as an additional module and more details can be found in Appendix C.
The procedure of generating multiple graph capsules is summarized as follows:
1) Scale primary capsules: Apply Attention Module to scale primary capsules.
The results of this module should be S ∈ R N × C all × d . 2)
Calculate votes: When calculating votes, capsules of different nodes from the same channel share the transform matrix.
The result of this step is a set of votes V ∈ R N × ( C all ) × P × d where C all denotes the number of channels.
P denotes the deﬁned number of graph capsules.
3) Dynamic Routing Mechanism: High-level graph capsules are computed with the procedure introduced in Section 2.3 based on votes produced in previous steps.
. C LASSIFICATION
This block is designed for graph classiﬁcation using the graph capsules.
Classiﬁcation Loss .
Dynamic Routing is applied again over graph capsules to generate ﬁnal class capsules C ∈ R K × d , where K is the number of graph classes.
Here, we use margin loss function proposed in Sabour et al.
(2017) to calculate the classiﬁcation loss
and it is computed as:
[Equation]
where m + = 0 .
9 , m − = 0 .
1 and T k =
iff the input graph belongs to class k .
λ is used to stop initial learning from reducing the length of all class capsules especially when K is large.
Reconstruction Loss .
Following Sabour et al.
(2017), we use reconstruction loss as regularization method.
Here, all class capsules are masked except the correct one and it is decoded with two fullyconnected layer to reconstruct the input information.
The information we reconstruct here is the histogram of input nodes.
The procedure can be written as:
[Equation]
where m i represents the number of nodes with the attribute i appear in the input graph, d i is the corresponding decoded value.
MP i =
iff input graph contains nodes with attribute i .
Equationis used to prevent reducing reconstruction loss from setting all decoded value as 0 especially when most of the elements of the ground truth are 0.
The architecture details presented in section 3.1 describe the key design idea of CapsGNN which is based on the fusing of GNN and CapsNet.
We also present a general comparison between CapsGNN with existing approaches in Appendix D.
E XPERIMENTS
We verify the performance of graph embeddings extracted from CapsGNN against a number of
SOTA approaches and some classical approaches on classiﬁcation task with 10 benchmark datasets.
Besides, we conduct experimental study to assess the impact of capsules in efﬁciency of encoding features of graphs.
We also conduct brief analysis on the generated graph/class capsules.
The experimental setting, results and analysis is shown in the following.
In addition to the analysis of the whole architecture, we also provide a comparison experiment to evaluate the contribution of each module of CapsGNN in classiﬁcation task and a brief analysis of learned attention distribution.
More details can be found in Appendix F and Appendix E respectively.
We will publish all of our experiment code in the near future.
4. D ATASET
Five biological graph datasets: MUTAG, ENZYMES, NCI1, PROTEINS, D&D and ﬁve social network datasets: COLLAB, IMDB-B, IMDB-M, RE-M5K, RE-M12K (Yanardag & Vishwanathan, 2015) are used for our experimental study.
Details of these datasets can be found in Appendix B.
4. B ASELINES M ETHODS
We compare CapsGNN with both kernel-based and deep-learning-based algorithms.
The details are given as follows:
Kernel-based Methods : Three kernel-based algorithms, namely the Weisfeiler-Lehman subtree kernel (WL) (Shervashidze et al., 2011), the graphlet count kernel(GK) (Shervashidze et al., 2009), and the Random Walk (RW)
(Vishwanathan et al., 2010).
Typically, kernel-based algorithms ﬁrst decompose graphs into sub-components based on the kernel deﬁnition, then build graph embeddings in a feature-based manner.
Lastly, some machine learning algorithms (i.
e., SVM) are applied to perform graph classiﬁcation.
Deep-Learning-based Methods : Three types of deep-learning-based algorithms are selected: 1) Graph2vec
(Narayanan et al., 2017), Deep Graph Kernel (DGK)(Yanardag & Vishwanathan, 2015) and AWE (Ivanov & Burnaev, 2018).
Graph2vec, DGK and AWE require extracting substructures in advance while Graph2vec and AWE learn the representation of graphs in the manner of
Doc2vec (Le & Mikolov, 2014), DGK applies Word2vec (Mikolov et al., 2013) to learn the similarity between each pair of sub-structures which will be used to build graph kernel.
Then kernel-based machine learning methods (i.e., SVM) are applied to perform graph classiﬁcation.
These three algorithms as well as kernel-based methods are all sub-components based which require computationally intensive preprocessing effort.
So although Graph2vec, DGK and AWE apply learning approaches to learn the embeddings, we still consider Graph2vec, DGK , AWE and other kernel-based algorithms as the same type in our experiments and we mainly compare our proposed architecture with the other remained methods which are all end-to-end and totally data-driven architectures.
2) PATCHY-SAN (PSCN)(Niepert et al., 2016).
This method ﬁrst sorts all nodes, then deﬁnes a receptive-ﬁeld size for each node.
These receptive-ﬁeld are then ﬁlled with sorted neighbor nodes.
Lastly, 1-D CNN is applied to perform graph classiﬁcation.
3) GCAPS-CNN (Verma & Zhang, 2018), Dynamic Edge CNN (ECC) (Simonovsky & Komodakis, 2017) and Deep Graph CNN (DGCNN)
(Zhang et al., 2018).
These methods are all GNN-based algorithms.
GCAPS-CNN ﬁrst extract FGSD (Verma & Zhang, 2017) features for nodes without attributes and then generate capsules for each node with higher-order statistical moment value of its neighbor nodes.
At the last layer, they calculate covariance between all nodes to generate graph embeddings.
ECC extracts node features on the condition of edge labels in GNN and then apply multi-scale pyramid structure to coarsen the graph.
It uses average pooling at the last layer to generate graph embeddings.
DGCNN generates nodes embeddings through a multi-layer GNN and combine features extracted from all layers.
Then they order the nodes based on the embeddings extracted from the last layer which is followed by 1-D CNN.
4. E XPERIMENTAL S ET - UP
The same architecture setting is used in CapsGNN for all datasets to show its robust performance.
For the node capsules extraction, the GNN has 5 layers ( L = 5 ), the number of channels at each layer is set as the same which is 2 ( C l = 2 ).
The number of graph capsules is ﬁxed as 16 ( P = 16 ).
The dimension of all capsules are set as 8 ( d = 8 ).
The number of units in the hidden layer of Attention
Module is set as 116 of the number of input units.
The number of iterations in routing is set as 3 .
During training stage, we simultaneously reduce Loss c and Loss r
and we scale Loss r with 0 .so that the model focuses on classiﬁcation task.
λ is set as 0 .
5 and 1 .
for multi-class classiﬁcation and binary classiﬁcation respectively.
Considering ENZYMES is a small dataset which containstypes of graphs and each type only contains 100 graphs, we add 20% noise to the input data when we train ENZYMES 2 .
To evaluate the performance objectively, we applied 10-fold cross validation on each dataset.
Each time we use 1 training fold as validation fold to adjust hyper-parameters, 8 training fold to train weighs and the remained 1 testing fold to test the performance.
We stop training when the performance on validate fold start to reduce.
Then we use the accuracy on test fold as our test result.
The ﬁnal result is the average of these 10 test accuracy.
By default, we use the results reported in their original work for baseline comparison.
However, in cases where the results are not available, we use the best testing results reported in Verma & Zhang (2018), Zhang et al.
and Ivanov &
Burnaev (2018).
4. C LASSIFICATION R ESULT
Table 1 lists the results of the experiments on biological datasets, Table 2 lists the results of the experiments on social datasets.
For each dataset, we highlight the top 2 accuracy in bold.
Compared with all the other algorithms, CapsGNN achieves top 2 on 6 out of 10 datasets and achieves comparable results on the other datasets.
Compared with all the other end-to-end architectures, CapsGNN achieves top 1 on all the social datasets.
[Table]
The proposed architecture achieves the SOTA performance on Social datasets.
More speciﬁcally, we are able to improve the classiﬁcation accuracy by a margin of 2.78% and 5.30% on RE-M5K and RE-M12K respectively.
This demonstrates that learning features in the form of capsules and [Table] modeling a graph to multiple embeddings is beneﬁcial to capture macroscopic properties of graphs.
These macroscopic properties are more important in classifying social networks.
These results also consistent with the property of CapsNet, as it focuses more on extracting important information from children capsules by voting.
However, applying routing to the whole graph leads to preserve all the information at a graph level and this property is not suitable to give prominence to individual ﬁne structures which might be more important to biological datasets analysis.
This results in less robust of CapsGNN on biological datasets.
Despite this, the performance of CapsGNN in graph classiﬁcation task still demonstrates its capability of graph representation especially its high potential of large graph dataset analysis.
4. E FFICIENCY OF C APSULES
The main objective of this experiment is to examine the efﬁciency of capsules in encoding graph features.
More efﬁcient in feature encoding here means representing more information with the similar number of neurons.
We construct a scalar-based neural network for each CapsGNN and then compare the CapsGNN with its related scalar-based architecture by comparing their training and testing accuracy on graph classiﬁcation task to demonstrate the efﬁciency in feature representation.
More speciﬁcally,these scalar-based architectures are designed by replacing the graph capsules block and the class capsules block in CapsGNN with fully-connected layers (FC).
In this case, the only difference between each pair of CapsGNN and its corresponding scalar-based architecture is that CapsGNN represents features with vectors and uses routing to propagate information between layers while the scalar-based architecture encodes features with scalar values.
In this experiment, the number of layers of GNN is set as 3 ( L = 3 ), the number of channels at each layer is all set as 2 ( C l = 2 ).
We construct different CapsGNNs by adjusting the dimension of nodes ( d n ) and graphs ( d g ) capsules and the number of graph capsules ( P ).
The size of FC in scalar-based architectures is adjusted based on the size of CapsGNNs so that they have comparable number of trainable weights.
Other hyper-parameters are the same as 4.3.
The details of the tested architectures are shown in Table 3.
Besides, NCI1 dataset, which has more than 4000 graphs, is used for the test.
The accuracy of NCI1 on various architectures can be found in Figure 3.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
8 ×16 ×2 ×4 ×8 ×16 ×
[Table]
[Table]
[Figure]
[Figure] [Figure] [Figure] In Table 3 and Figure 3, the setting of different architectures is represented as d n - d g - P .
Here, we choose the simplest setting ( 2-4-2 ) as an example: 2-4-2 means that the dimension of nodes capsules is d n = 2 , the dimension of graph and class capsules is
d g = 4 and the number of graph capsules
P equals to 2 .
Besides, we set the dimension of FC of its corresponding scalar-based architecture as 12 so that they have comparable number of trainable weights.
In this case, each graph is modeled as 2 4 -dimensional graph embeddings in the CapsGNN or 1 12 -dimensional graph embedding in its corresponding scalar-based architecture.
Both architectures are sub-optimal to represent the whole dataset while CapsGNN can still reach higher accuracy compared with scalar-based architecture.
As we can see from Figure 3, the test accuracy of Caps-based architectures (CapsGNN) is higher than the corresponding scalar-based architectures in all settings.
For the training accuracy, when the dimension of FC is slightly higher than the dimension of graph embeddings in CapsGNN, CapsGNN can still reach higher accuracy which indicates that CapsGNN is more powerful in representing the whole dataset.
When we keep increasing the number of graph capsules in CapsGNN and enlarging the dimension of FC in scalar-based architectures, the difference between the dimension of graph embeddings and the size of FC becomes larger, their training accuracy will be closer.
It is noted that the training accuracy of scalar-based architectures is slightly higher than the CapsGNNs when the dimension of FC is about 20% larger than the dimension of graph capsules.
In this experiment, we use extremely simple architectures on purpose to simulate the situation where we need to model complex datasets with relatively simple architectures.
Since each pair of CapsGNN and its corresponding scalar-based architecture have similar structure and comparable number of trainable weights, the higher training accuracy and testing accuracy of CapsGNN demonstrate its efﬁciency in feature encoding and its strong capability of generalization.
4. G RAPH C APSULES A NALYSIS
CapsGNN leverages on capsules idea to get multiple embeddings for each graph so that complex information underlying graphs can be captured more effectively.
To explore the properties of the extracted graph capsules and class capsules, we plot the graph distribution based on capsules extracted from different channels with t-SNE.
Due to space constrain, we only take Reddit-12K as an example.
We choose to depict the distribution of graphs which are generated from 3 categories, namely atheism , IAmA and mildlyinteresting with capsules extracted from the 1st, 2nd, 11th, 14th channel of graph capsules.
As we can see from Table 4, different channels of capsules represent different aspects of graph properties.
atheism and IAmA can be discriminated obviously with capsules extracted from the 11th and the 14th channels while they are hard to be separated with capsules extracted from the 1st and the 2nd channels.
However, atheism and mildlyinteresting can be discriminated with the capsules extracted from the 1st and the 2nd channels while they are mixed in the 11th and the 14th channels which is opposite to the case of atheism and IAmA .
This phenomenon can also be [Table] observed in other multi-class datasets.
It is still hard to ﬁgure out the speciﬁc aspects these capsules focus on.
However, compared with scalar-based neural networks, modeling an object with multiple embeddings makes it possible to explore the meaning of each channel which may lead the model to learn more interpretable embeddings in the future.
[Table] As we can see from Table 5, different class capsules focus on different classiﬁcation-related graph properties.
For example, the capsules that represent athesism (ﬁrst column) can well discriminate athesism (red) from the other two types of graphs while IAmA (green) and mildlyinteresting (blue) are mixed in this channel.
The similar phenomenon can also be found in other class capsules.
Besides, when we concatenate the capsules of these three classes together, three types of graphs can be well discriminated with the concatenated capsules which also directly reﬂect the classiﬁcation performance.
This property is quite different from standard scalar-based architectures where each graph is modeled with only one graph embedding.
By introducing the concept of capsules, the graph and class capsules can not only preserve classiﬁcation-related properties of each graph (reﬂected with the length of class capsules) but also other properties information (reﬂected with the angle of class capsules).
The generated class capsules can also be useful in other follow-up work and we leave this to be explored in the future.
C ONCLUSION
We have proposed CapsGNN, a novel framework that fuse capsules theory into GNN for more efﬁcient graph representation learning.
Inspired by CapsNet, the concepts of capsules are introduced in this architecture to extract features in the form of vectors on the basis of nodes features extracted from GNN.
As a result, one graph is represented as multiple embeddings and each embedding captures different aspects of the graph properties.
The generated graph and class capsules can preserve not only the classiﬁcation-related information but also other information with respect to graph properties which might be useful in the follow-up work and we leave this to be explored in the future.
This is a novel, efﬁcient and powerful data-driven method to represent high-dimensional data such as graphs.
Our model has successfully achieved better or comparable performance when compared with other SOTA algorithms on 6 out of 10 graph classiﬁcation tasks especially on social datasets.
Compared with similar scalar-based architectures, CapsGNN is more efﬁcient in encoding features and this would be very beneﬁcial for processing large datasets
A S PECIFIC PROCEDURE OF ROUTING
The speciﬁc procedure of routing is shown in Algorithm 1.
1: procedure D YNAMIC R OUTING ( t, S , W ) 2: for all children capsule i : v j | i =
s Ti W ij 3: for all children capsule i to all parent capsule j : r ij ←4: for t iterations do 5: for all children capsule i :
˜ r i ← softmax ( r i ) 6: for all parent capsule j : h j ← (cid:80) i ˜
r ij v ij 7: for all parent capsule j : ˜ h
j ← squash ( h j ) 8: for all children capsule i to all parent capsule j : r ij ← r ij + ˜ h
Tj v ij 9: end for 10: return ˜ h
j 11: end procedure
B D ETAILS OF EXPERIMENTAL DATASETS
The details of benchmark datasets we use in our experiment is shown in Figure 6.
[Table]
C C OORDINATE A DDITION
After Attention Module, coordinate addition can be used to preserve the position information of each node during the procedure of generating node capsule votes.
The details of Coordinate Addition module can be found in Figure 4.
This module is not necessary in some datasets.
Here, we propose this module just for an optimization module.
When the GNN goes deeper, the extracted nodes features contain more speciﬁc position information of each node.
Inspired by Zhang et al.
(2018) where the node embeddings learned from the last layer of GNN are taken to order all node, we also take the capsules extracted from the last layer of GNN as the position indicators of corresponding nodes by concatenating it with each capsule of the node.
The procedure of calculating votes with node position indicators can be written as:
[Equation]
where v ( n,i ) j ∈ R 1 × ( d n + d p ) represents the node capsule vote from the i th channel of the n th node to the j th channel of parent capsules.
W nij ∈ R
d × d n and W pj ∈ R d × d p are the transform matrix.
s ( n,i ) is the same as introduced in Section 3.1.2 and (cid:107) represents concatenate operation.
D D IFFERENCES FROM E XISTING A PPROACHES
Here, we present a general comparison between CapsGNN with existing approaches.
1) Compared with Atwood & Towsley (2016), Simonovsky & Komodakis (2017) and Zhang et al.
(2018) (GNN-based graph representation learning architectures), CapsGNN represents node features in the form of capsules.
This is helpful to preserve the properties information contained in nodes more efﬁciently when generating graph embeddings.
Besides, each graph is modeled as multiple embeddings in CapsGNN instead of only one embedding used in other approaches.
This allows us to capture information of graphs from different aspects.
The second difference is that, in these approaches, each part of the graph is given equal importance.
However, the attention mechanism used in CapsGNN allows it to assign various weighting to different nodes.
This leads the model to focus more on critical parts of input graphs.
Lastly, different from Atwood & Towsley (2016) and
Simonovsky & Komodakis (2017), CapsGNN and Zhang et al.
(2018) uses node features extracted from multiple layers of GNN so that different size of receptive-ﬁelds are applied to preserve more information.
2) GCAPS-CNN proposed by Verma & Zhang (2018) also introduced capsules-related concept into graph representation learning.
However, they generate capsules in a feature-based manner instead of learning capsules as distributed embeddings.
More speciﬁcally, when they extend a scalar feature to a capsule for the node n , P higher-order statistical moment value is calculated based on its neighbor nodes and this P value is concatenated to a P -dimensional capsule.
Between layers, GCAPSCNN performs dimension reduction to compress capsules back to scalar features, which defeats the purpose of having capsules in the ﬁrst place.
CapsGNN learns each dimension of capsules in a data-driven manner and apply routing mechanism between layers to preserve the learned meaning of each capsule.
This also allows us to preserve multiple properties information contained in nodes more efﬁciently especially when generating graph embeddings.
Compared with CapsNet proposed by Sabour et al.
(2017) which works well in image processing domain, CapsGNN needs to handle more complex situations when handling graphs.
In image processing domain, the size of the input images can be standardized by resizing the images.
However, it is not possible to simply resize the graphs.
So, we introduced an additional Attention Module to tackle graphs that are vastly different in sizes and preserve important parts of graphs.
We also propose to use features extracted from all layers of GNN since it is hard to deﬁne a suitable receptive-ﬁeld size for graphs.
Furthermore, compared with the architecture of CapsNet, CapsGNN has one additional graph capsules layer which is used to learn multiple graph embeddings and these embeddings reﬂect different aspects of graph properties which is valuable in future research.
To the best of our knowledge, we are the ﬁrst one to model a graph as multiple embeddings in the form of distributed capsules and we believe this approach of learning representations has a high potential for other complex data analysis which is not limited to graphs.
Besides, CapsGNN has different explanation of linear transformation.
In CapsNet, by applying a linear trainable transformation to pose vectors, the spatial relationship between object parts and the whole object can be well modeled.
However, by applying linear trainable transformation, CapsGNN is simply computing the prediction vectors from nodes-level representations to graph-level representations.
This transform matrix is not trying to model the change of viewpoint or capture viewpoint invariant knowledge but to model the relationship between the properties of nodes and the properties of the whole graph.
E A TTENTION A NALYSIS
We conduct node-based normalization in Attention Module to scale different node capsules to lead the model focusing on critical parts of each graph.
In this experiment, we plot the attention distribution on REDDIT and IMDB-M dataset to see if this model indeed focuses on critical parts of graphs.
Graphs in IMDB-M are all ego-networks derived from movie-collaboration relationship networks in which each node represents an actor/actress and there is a link between two nodes if they appeared in the same movie.
These collaboration networks are built based on movies in different genres namely
Romance , Sici-Fi and Comedy and the task is to predict where the given graph belongs.
As for REDDIT, each graph in REDDIT dataset represents a discussion thread in which each node indicates a user and there is a link between two users if at least one of them responded to another’s comment.
The task is to predict the subreddit where the given discussion graph belongs.
[Figure]
The visualization results are shown in Figure 5 where we can ﬁnd that the model always pays more attention to nodes with high centrality or critical nodes that connect multiple separate subcomponents of the graph which makes sense as critical parts of graphs from common sense.
Beside, from the results in Table 7, we can ﬁnd that the Attention Module indeed beneﬁts to the classiﬁcation accuracy in some datasets.
F C ONTRIBUTION OF E ACH M ODULE
In addition to evaluate the performance of the whole architecture on classiﬁcation task, we also provide detailed study to quantify the contributions made by each module in CapsGNN on classiﬁcation task.
The four comparison architectures Set-up is shown as below and the setting of the hyper-parameters are the same as 4.3.
In this experiment, we also add 20% noise to input data for all datasets.
1) GNN + Attention + Routing + Reconstruction + Margin Loss (CapsGNN) : Basic CapsGNN. 2) GNN + Average + Routing + Reconstruction + Margin Loss (CapsGNN-Avg) : The Attention module in basic CapsGNN is replaced with the Average module.
3) GNN + Attention + Reconstruction + Margin Loss (CapsGNN-noRout) :
In this architecture, we will set the similarity between all the capsules from one layer to the next layer as the same so that each children capsule will be equally routed to all the parent capsules.
4) GNN + Attention + Routing + Margin Loss (CapsGNN-noRecon) :
In this architecture, we directly remove the Reconstruction loss module.
[Table]
As we can see from Table
[Table]
each module has different contribution in different datasets.
When we remove Dynamic Routing (CapsGNN-noRout), the accuracy of the whole architecture is reduced by 0.90% for PROTEINS and 1.7% for NCI1 while keeps similar in IMDB-M and IMDB-B and COLLAB.
The comparison between CapsGNN with CapsGNN-Avg indicates that the Attention Module is only helpful to the classiﬁcation accuracy in some datasets such as PROTEINS where the accuracy is improved by 2.24%.
In COLLAB and IMDB-B, replacing Attention Module with normal average operation can even reach higher accuracy.
Reconstruction Module is an regularization module in CapsGNN and removing this module causes a lower classiﬁcation accuracy on smaller datasets such as IMDB-M, IMDB-B and PROTEINS where the accuracy is reduced by 1.1%, 1.2% and 1.4% respectively.
Overall, CapsGNN is a general framework that fuse capsule theory to GNN for more efﬁcient graph representation learning.
In this framework, we also provide multiple possible modules to improve the quality of learned graph embeddings while we do not target to ﬁnd the best combination of modules for each dataset.
Since each possible module plays a different role in different datasets, it would be better to adjust the architecture and hyper-parameters based on practical situation.