C APSULE G RAPH N EURAL N ETWORK
A BSTRACT
The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance.
However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representations may not sufﬁce to preserve the node/graph properties efﬁciently , resulting in sub-optimal graph embeddings.
Inspired by the Capsule Neural Network (CapsNet) (Sabour et al., 2017), we pro - pose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings al gorithms
By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level
As a result , our model generates multiple embeddings for each graph to capture graph properties from different aspects.
The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs
Our extensive evaluations with 10 graph-structured datasets demonstrate that
CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven.
It outperforms other SOTA techniques on several graph classiﬁcation tasks, by virtue of the new instrument.
I NTRODUCTION
GNN is a general type of deep-learning architectures that can be directly applied to structured data.
These architectures are mainly generalized from other well-established deep-learning models like
CNN (Krizhevsky et al., 2012) and RNN (Mikolov et al., 2010).
In this paper, we mainly focus on Convolution-based Graph Neural Networks which attract increasing interest recently.
Convolution operation can be embedded into Graph Neural Networks from spectral or spatial perspective.
Bruna et al.
(2013) deﬁnes the convolution operation in the Fourier domain which needs to calculate the eigendecomposition of the graph Laplacian.
This method is computationally expensive and the ﬁlters they deﬁned are non-spatially localized.
Later, Henaff et al.
(2015) introduces Chebyshev expansion of the graph Laplacian to avoid computing eigenvectors and Kipf & Welling (2017) proposes to do convolution within 1-step neighbor nodes to reduce the complexity.
From the spatial perspective, Hamilton et al.
(2017) and Zhang et al.
(2018) propose to deﬁne a node receptive-ﬁeld and do convolution within this ﬁeld during which the information of each node as well as their neighbor nodes is gathered and new representation of each node is generated through an activation function.
Both of these two perspectives perform well in node representation learning and a number of variants (Velikovi et al., 2018) are developed based on the convolution idea and some of them have proven to achieve SOTA in various tasks.
The success of GNN in node representation learning has inspired many deep-learning-based approaches to leverage on node embeddings extracted from GNN to generate graph embeddings for graph-based applications.
However, during this procedure, the learned representation of each node will be considered as multiple individual scalar features instead of one vector.
For example, Zhang et al.
(2018) applies element-wise max-pooling to nodes embeddings when generating graph embeddings, Verma & Zhang (2018) generates graph embeddings by computing the element-wise covariance of all nodes.
These operations indicate that the authors capture node features in the form of scalar when they generate graph embeddings which may not sufﬁce to preserve the node/graph properties efﬁciently.
To build high-quality graph embeddings, it is important to not only detect the presence of different structures around each node but also preserve their detailed properties such as position, direction, connection, etc.
However, encoding these properties information in the form of scalar means activating elements in a vector one-by-one which is exponentially less efﬁcient than encoding them with distributed representations.
This has been identiﬁed discussed in Sabour et al.
Inspired by CapsNet, we propose to extend scalar to vector during the procedure of applying GNN to graph representation learning.
Compared with scalar-based neural network, vector-based neural network preserves the information of node/graph properties more efﬁciently.
The technique for extracting features in the form of vectors is proposed in Hinton et al.
and improved in Sabour et al.
(2017) and Hinton et al.
This technique is mainly devised for image processing.
In their work, the extracted vector is referred to as capsule (a group of neurons in neural network), so we follow the same notation in our work.
Introducing capsules allows us to use routing mechanism to generate high-level features which we believe is a more efﬁcient way for features encoding.
Compared with max-pooling in CNN in which all information will be dropped except for the most active one, routing preserves all the information from low-level capsules and routes them to the closest high-level capsules.
Besides, this allows to model each graph with multiple embeddings and each embedding reﬂects different properties of the graph.
This is more representative than only one embedding used in other scalar-based approaches.
In this paper, we propose Capsule Graph Neural Network (CapsGNN), a novel deep learning architecture, which is inspired by CapsNet and uses node features extracted from GNN to generate high-quality graph embeddings.
In this architecture, each graph is represented as multiple embeddings and each embedding reﬂects the graph properties from different aspects.
More speciﬁcally, basic node features are extracted in the form of capsules through GNN and routing mechanism is applied to generate high-level graph capsules as well as class capsules.
In the procedure of generating graph capsules, an Attention Module can be applied to tackle graphs in various sizes.
It also assigns different weights to each capsule of each node so that this model focuses on critical parts of the graph.
We validate the performance of generated graph embeddings on classiﬁcation task over 5 biological datasets and 5 social datasets.
CapsGNN achieves SOTA performance on 6 out ofbenchmark datasets and comparable results on the rest.
T-SNE (Maaten & Hinton, 2008) is used to visualize the learned graph embeddings and the results show that different graph capsules indeed capture different information of the graphs.
B ACKGROUND
Here, we provide a brief introduction to Graph Convolutional Networks (GCNs) (Kipf & Welling, 2017), routing mechanism in CapsNet and Attention mechanism which is used in CapsGNN.
2. G RAPH
By deﬁnition, a weighted directed graph can be represented by G =
( V , X , A ) where V = { v 1 , v 2 , ...v N } is the set of nodes and A ∈ { 0 , 1 } N × N is the adjacency matrix.
If there is an edge from v i to v j , then A ij = 1 otherwise A ij = 0 .
X ∈ R N × d represents the features of each node. d is the number of feature channels and N is the number of nodes.
2. G RAPH C ONVOLULTIONAL N ETWORK
GCN, a widely used GNN architecture, is chosen as one of the key building blocks in our work.
At each layer of the GCN, the convolution operation is applied to each node as well as its neighbors and the new representation of each node is computed through an activation function.
This procedure can be written as:
[Equation]
where Z l ∈ R N × d represents nodes features at the layer l , d represents the number of feature channels and Z
=
X , W l ∈ R d × d (cid:48) is a trainable weights matrix which serves as a channel ﬁlter, f is a nonlinear activation function, T ∈ R N × N is the information transform matrix and it is usually calculated from the adjacency matrix A for guiding the information ﬂowing between nodes.
A complete GNN usually stacks L layers to generate ﬁnal nodes embeddings Z L .
In the architecture proposed by Kipf & Welling (2017), at the l th layer of GCN, the extracted features of each node actually take all its adjacent nodes within l steps into consideration.
So l can be considered as the size of the node receptive-ﬁeld at this layer.
This special property inspired us to use nodes features extracted from different layers to generate the graph capsules.
2. C APSULE N EURAL N ETWORK
The concept of capsules is invented by Hinton’s team (Hinton et al., 2011) and used recently in
Sabour et al.
(2017) and Hinton et al.
CapsNet is designed for image features extraction and it is developed based on CNN.
However, unlike traditional CNN in which the presence of feature is represented with scalar value in feature maps, the features in CapsNet are represented with capsules (vectors).
In Sabour et al.
(2017), the direction of capsules reﬂects the detailed properties of the features and the length of capsules reﬂects the probability of the presence of different features.
The transmission of information between layers follows Dynamic Routing mechanism.
The speciﬁc procedure of Dynamic Routing can be found in Appendix A for the completeness.
Inspired by CapsNet, the capsule mechanism is adopted and fused with GNN in our proposed CapsGNN to generate graph capsules and class capsules on the basis of node capsules which are extracted from GNN.
Dynamic Routing is applied to update weights between capsules from one layer to the next layer so that the properties captured by node capsules can be propagated to suitable graph capsules.
Thus, each graph is modeled as multiple graph capsules, and then modeled as multiple class capsules.
Different graph capsules reﬂect the properties of the graph from different aspects.
2. A TTENTION M ECHANISM
Attention mechanism is widely applied in image (Zheng et al., 2017) and natural language processing domain (Gehring et al., 2016) where it is used to ﬁnd the relevant parts of the input data to the task target.
The main procedure of Attention mechanism is: 1) deﬁning an attention measure which is used to measure the relevance of each part of the input data to the task target.
2) normalizing the generated attention value.
3) scaling each part with the normalized attention value.
In CapsGNN, we apply Attention mechanism for two purposes: 1) scaling each node capsule so that the graph capsules that are generated from different graphs are still comparable even though these graphs are vastly different in sizes.
2) guiding the model to focus on more relevant parts of graphs.
C APSULE G RAPH N EURAL N ETWORK
In this section, we outline CapsGNN and show how it is used to generate high-quality graph capsules which then can be applied to graph classiﬁcation task.
Figure 1 shows a simpliﬁed version of
CapsGNN.
It consists of three key blocks: 1) Basic node capsules extraction block: GNN is applied to extract local vertices features with different receptive-ﬁeld and then primary node capsules are built in this block.
2) High level graph capsules extraction block: Attention Module and Dynamic Routing are fused to generate multiple capsules for graphs.
3) Graph classiﬁcation block: Dynamic Routing is applied again to generate class capsules for graph classiﬁcation.
The details of each block is explained in the following.
B ASIC N ODE C APSULES
Firstly, the basic node features are extracted with GNN.
Node degrees can be used as node attributes if nodes do not have attributes.
We use the architecture improved by Kipf & Welling (2017) (GCN) as the node features extractor.
The difference is that we extract multi-scale node features from different layers and the extracted features are represented in the form of capsules.
The procedure can be written as:
[Equation]
where W lij ∈ R d × d (cid:48) is the trainable weights matrix.
It serves as the channel ﬁlters from the i th channel at the l th layer to the j th channel at the ( l + 1) th layer.
Here, we choose f ( · ) =
tanh ( · ) as the activation function.
Z l +1 ∈ R N × d (cid:48) , Z 0 = X , ˜
A =
A + I 1 and ˜ D
= (cid:80)
j ˜
A ij .
To preserve features sub-components different sizes, we use nodes features
GNN layers high-level capsules.
3. H IGH -L EVEL G RAPH C APSULES
After getting local node capsules, global routing mechanism is applied to generate graph capsules.
The input of this block contains N sets of node capsules, each set is S n = { s 11 , .., s 1 C 1 , ..., s LC L } , s lc ∈ R d , where C l is the number of channels at the l th layer of GNN, d is the dimension of each capsule.
The output of this block is a set of graph capsules H ∈ R P × d (cid:48) .
Each of the capsules reﬂects the properties of the graph from different aspects.
The length of these capsules reﬂects the probability of the presence of these properties and the angle reﬂects the details of the graph properties.
Before generating graph capsules with node capsules, an Attention Module is introduced to scale node capsules.
Attention Module .
In CapsGNN, primary capsules are extracted based on each node which means the number of primary capsules depends on the size of input graphs.
In this case, if the routing mechanism is directly applied, the value of the generated high-level capsules will highly depend on the number of primary capsules (graph size) which is not the ideal case.
Hence, an Attention Module is introduced to combat this issue.
The attention measure we choose is a two-layer fully connected neural network F attn ( · ) .
The number of input units of F attn ( · ) is d × C all where C all = (cid:80) l C l and the number of output units equals to C all .
We apply node-based normalization to generate attention value in each channel and then scale the original node capsules.
The details of Attention Module is shown in Figure 2 and the procedure can be written as:
[Equation]
where ˜ s n ∈ R 1 × C all d is obtained by concatenating all capsules of the node n .
s ( n,i ) ∈
R 1 × d represents the i th capsule of the node n and F attn ( ˜ s n ) ∈ R 1 × C all is the generated attention value.
In this way, the generated graph capsules can be independent to the size of graphs and the architecture will focus on more important parts of the input graph.
After Attention Module, coordinate addition module can be used to preserve the position information of each node during the procedure of generating node capsule votes.
Here, we introduce coordinate addition as an additional module and more details can be found in Appendix C.
The procedure of generating multiple graph capsules is summarized as follows:
1) Scale primary capsules: Apply Attention Module to scale primary capsules.
The results of this module should be S ∈ R N × C all × d . 2)
Calculate votes: When calculating votes, capsules of different nodes from the same channel share the transform matrix.
The result of this step is a set of votes V ∈ R N × C all × P × d where C all denotes the number of channels.
P denotes the deﬁned number of graph capsules.
3) Dynamic Routing Mechanism: High-level graph capsules are computed with the procedure introduced in Section 2.3 based on votes produced in previous steps.
3. C LASSIFICATION
This block is designed for graph classiﬁcation using the graph capsules.
Classiﬁcation Loss .
Dynamic Routing is applied again over graph capsules to generate ﬁnal class capsules C ∈ R K × d , where K is the number of graph classes.
Here, we use margin loss function proposed in Sabour et al.
(2017) to calculate the classiﬁcation loss
and it is computed as:
[Equation]
where m + = 0 .
9 , m − = 0 .
1 and T k =
iff the input graph belongs to class k .
λ is used to stop initial learning from reducing the length of all class capsules especially when K is large.
Reconstruction Loss .
Following Sabour et al.
(2017), we use reconstruction loss as regularization method.
Here, all class capsules are masked except the correct one and it is decoded with two fullyconnected layer to reconstruct the input information.
The information we reconstruct here is the histogram of input nodes.
The procedure can be written as:
[Equation]
where m i represents the number of nodes with the attribute i appear in the input graph, d i is the corresponding decoded value.
MP i =
iff input graph contains nodes with attribute i .
Equationis used to prevent reducing reconstruction loss from setting all decoded value as 0 especially when most of the elements of the ground truth are 0.
The architecture details presented in section 3 describe the key design idea of CapsGNN which is based on the fusing of GNN and CapsNet.
We also present a general comparison between CapsGNN with existing approaches in Appendix D.
E XPERIMENTS
We verify the performance of the graph embeddings extracted from CapsGNN against a number of
SOTA approaches and some classical approaches on classiﬁcation task with 10 benchmark datasets.
Besides, we conduct experimental study to assess the impact of capsules in efﬁciency of encoding features of graphs.
We also conduct brief analysis on the generated graph/class capsules.
The experimental results and analysis is shown in the following.
In addition to the analysis of the whole framework, we also provide a comparison experiment to evaluate the contribution of each module of CapsGNN with classiﬁcation task.
More details can be found in Appendix F.
4. G RAPH C LASSIFICATION
The goal of graph classiﬁcation is to predict the classes these graphs belong to by analyzing the structure and nodes labels information of graphs.
More speciﬁcally, given a set of labeled graphs D = { ( G 1 , y 1 ) , ( G 2 , y 2 ) , . . .
} where y i ∈ Y is the label of each graph G i .
The objective of graph classiﬁcation is to ﬁnd a mapping f such that f : G → Y .
4.1. B ASELINES M ETHODS
We compare CapsGNN with both kernel-based and deep-learning-based algorithms.
The details are given as follows:
Kernel-based Methods : Three kernel-based algorithms, namely the Weisfeiler-Lehman subtree kernel (WL) (Shervashidze et al., 2011), the graphlet count kernel(GK) (Shervashidze et al., 2009), and the Random Walk (RW)
(Vishwanathan et al., 2010).
Typically, kernel-based algorithms ﬁrst decompose graphs into sub-components based on the kernel deﬁnition, then build graph embeddings in a feature-based manner.
Lastly, some machine learning algorithms (i
e., SVM) are applied to perform graph classiﬁcation.
Deep-Learning-based Methods : Three types of deep-learning-based algorithms are selected: 1) Graph2vec
(Narayanan et al., 2017), Deep Graph Kernel (DGK)(Yanardag & Vishwanathan, 2015) and AWE (Ivanov & Burnaev, 2018).
Graph2vec, DGK and AWE require extracting substructures in advance while Graph2vec and AWE learn the representations of graphs in the manner of Doc2vec (Le & Mikolov, 2014), DGK applies Word2vec (Mikolov et al., 2013) to learn the similarity between each pair of sub-structures which will be used to build the graph kernel.
Then kernel-based machine learning methods (i.e., SVM) are applied to perform graph classiﬁcation.
These three algorithms as well as kernel-based methods are all sub-components based and they all require two stages to do graph classiﬁcation.
So although Graph2vec, DGK and AWE apply learning approaches to learn the embeddings, we still consider them and other kernel-based algorithms as the same type in our experiments and we mainly compare our proposed architecture with the other remained methods which are all end-to-end and totally data-driven architectures.
2) PATCHY-SAN (PSCN)(Niepert et al., 2016).
This method ﬁrst sorts all nodes, then deﬁnes a receptive-ﬁeld size for each node.
These receptive-ﬁeld are then ﬁlled with sorted neighbor nodes.
Lastly, 1-D CNN is applied to perform graph classiﬁcation.
3) GCAPS-CNN (Verma & Zhang, 2018), Dynamic Edge CNN (ECC) (Simonovsky & Komodakis, 2017) and Deep Graph CNN (DGCNN)
(Zhang et al., 2018).
These methods are all GNN-based algorithms.
GCAPS-CNN ﬁrst extract FGSD (Verma & Zhang, 2017) features for nodes that do not have attributes and then generate capsules for each node with higher-order statistical moment value of its neighbor nodes.
At the last layer, they calculate covariance between all nodes to generate graph embeddings.
ECC extracts node features on the condition of edge labels in GNN and then apply multi-scale pyramid structure to coarsen the graph.
It uses average pooling at the last layer to generate graph embeddings.
DGCNN generates nodes embeddings through a multi-layer GNN and combine features extracted from all layers.
Then they order the nodes based on the embeddings extracted from the last layer which is followed by 1-D CNN.
4.1. E XPERIMENTAL S ET - UP
Five biological graph datasets: MUTAG, ENZYMES, NCI1, PROTEINS, D&D and ﬁve social network datasets: COLLAB, IMDB-B, IMDB-M, RE-M5K, RE-M12K (Yanardag & Vishwanathan, 2015) are used for our experimental study.
Details of these datasets can be found in Appendix B.
[-5 and 1 . 
-]
We applied 10-fold cross validation to evaluate the performance objectively.
Each time we usetraining fold as validation fold to adjust hyper-parameters, 8 training fold to train the architecture and the remained 1 testing fold to test the performance.
We stop training when the performance on the validation fold reaches to the highest.
Then we use the accuracy on the test fold as our test result.
The ﬁnal result is the average of these 10 test accuracy.
By default, we use the results reported in the original work for baseline comparison.
However, in cases where the results are not available, we use the best testing results reported in Verma & Zhang (2018), Zhang et al.
and Ivanov & Burnaev (2018).
More details about experimental setting can be found in Appendix E.
4.1. C LASSIFICATION R ESULT
Table 1 lists the results of the experiments on biological datasets, Table 2 lists the results of the experiments on social datasets.
For each dataset, we highlight the top 2 accuracy in bold.
Compared with all the other algorithms, CapsGNN achieves top 2 on 6 out of 10 datasets and achieves comparable results on the other datasets.
Compared with all the other end-to-end architectures, CapsGNN achieves top 1 on all the social datasets.
[Table]
CapsGNN achieves the SOTA performance on social datasets.
More speciﬁcally, we are able to improve the classiﬁcation accuracy by a margin of 2.78% and 5.30% on RE-M5K and RE-M12K respectively.
This demonstrates that learning features in the form of capsules and modeling a graph to multiple embeddings is beneﬁcial to capture macroscopic properties of graphs which are more important in classifying social networks.
These results also consistent with the property of CapsNet, as it focuses more on extracting important information from children capsules by voting.
However, applying routing to the whole graph leads to preserve all the information at a graph level and this property is not suitable to give prominence to individual ﬁne structures which might be more important to biological datasets analysis.
This results in less robust of CapsGNN on biological datasets.
Despite this, the performance of CapsGNN in graph classiﬁcation task still demonstrates its capability of graph representation especially its high potential of large graph dataset analysis.
[Table]
4. E FFICIENCY OF C APSULES
The main objective of this experiment is to examine the efﬁciency of capsules in encoding graph features.
More efﬁcient in feature encoding here means representing more information with the similar number of neurons.
We construct a scalar-based neural network for each CapsGNN and then compare the CapsGNN with its related scalar-based architecture by comparing their training and testing accuracy on graph classiﬁcation task to demonstrate the efﬁciency in feature representation.
More speciﬁcally,these scalar-based architectures are designed by replacing the graph capsules block and the class capsules block in CapsGNN with fully-connected layers (FC).
In this case, the only difference between each pair of CapsGNN and its corresponding scalar-based architecture is that
CapsGNN represents features with vectors and uses routing to propagate information between layers while the scalar-based architecture encodes features with scalar values.
In this experiment, the number of layers of GNN is set as L = 3 , the number of channels at each layer is all set as C l = 2 .
We construct different CapsGNNs by adjusting the dimension of nodes ( d n ) and graphs ( d g ) capsules and the number of graph capsules ( P ).
The size of FC in scalar-based architectures is adjusted based on the size of CapsGNNs so that they have comparable number of trainable weights.
Other hyper-parameters are the same as Appendix E.
The details of the tested architectures are shown in Table 3.
Besides, NCI1 dataset, which has more than 4000 graphs, is used for the test.
The accuracy of NCI1 on various architectures can be found in Figure 3.
[Table]
In Table 3 and Figure 3, the setting of different architectures is represented as d n - d g - P .
Here, we choose the simplest setting ( 2-4-2 ) as an example: 2-4-2 means that the dimension of nodes capsules is d n = 2 , the dimension of graph and class capsules is
d g = 4 and the number of graph capsules
P equals to 2 .
Besides, we set the dimension of FC of its corresponding scalar-based architecture as 12 so that they have comparable number of trainable weights.
In this case, each graph is modeled as 2 4 -dimensional graph embeddings in the CapsGNN or 1 12 -dimensional graph embedding in its corresponding scalar-based architecture.
Both architectures are sub-optimal to represent the whole dataset while CapsGNN can still reach higher accuracy compared with the scalar-based architecture.
As we can see from Figure 3, the test accuracy of Caps-based architectures (CapsGNN) is higher than the corresponding scalar-based architectures in all settings.
For the training accuracy, when the [Figure] dimension of FC is slightly higher than the dimension of graph embeddings in CapsGNN, CapsGNN can still reach higher accuracy which indicates that CapsGNN is more powerful in representing the whole dataset.
When we keep increasing the number of graph capsules in CapsGNN and enlarging the dimension of FC in scalar-based architectures, the difference between the dimension of graph embeddings and the size of FC becomes larger, their training accuracy will be closer.
It is noted that the training accuracy of scalar-based architectures is slightly higher than the CapsGNNs when the dimension of FC is about 20% larger than the dimension of graph capsules.
In this experiment, we use extremely simple architectures on purpose to simulate the situation where we need to model complex datasets with relatively simple architectures.
Since each pair of CapsGNN and its corresponding scalar-based architecture have similar structure and comparable number of trainable weights, the higher training accuracy and testing accuracy of CapsGNN demonstrate its efﬁciency in feature encoding and its strong capability of generalization.
4. G RAPH C APSULES A NALYSIS
CapsGNN leverages on capsules idea to get multiple embeddings for each graph so that complex information underlying graphs can be captured more effectively.
To explore the properties of the extracted graph/class capsules, we plot the graph distribution based on capsules extracted from different channels with t-SNE.
Due to space constrain, we only take REDDIT-M12K as an example.
Graph Capsules
Subreddit
Channel
[Table]
atheism & IAmA atheism & mildlyinteresting We choose to depict the distribution of graphs which are generated from 3 categories, namely atheism , IAmA and mildlyinteresting with capsules extracted from the 1st, 2nd, 11th, 14th channel of graph capsules.
As we can see from Table 4, different channels of capsules represent different aspects of graph properties.
atheism and IAmA can be discriminated obviously with capsules extracted from the 11th and the 14th channels while they are hard to be separated with capsules extracted from the 1st and the 2nd channels.
However, atheism and mildlyinteresting can be discriminated with the capsules extracted from the 1st and the 2nd channels while they are mixed in the 11th and the 14th channels which is opposite to the case of atheism and IAmA .
This phenomenon can also be observed in other multi-class datasets.
It is still hard to ﬁgure out the speciﬁc aspects these capsules focus on.
However, compared with scalar-based neural networks, modeling an object with multiple embeddings makes it possible to explore the meaning of each channel which may lead the model to learn more interpretable embeddings in the future.
[Table] As we can see from Table 5, different class capsules focus on different classiﬁcation-related graph properties.
For example, the capsules that represent athesism (ﬁrst column) can well discriminate athesism (red) from the other two types of graphs while IAmA (green) and mildlyinteresting (blue) are mixed in this channel.
The similar phenomenon can also be found in other class capsules.
Besides, when we concatenate the capsules of these three classes together, three types of graphs can be well discriminated with the concatenated capsules which also directly reﬂect the classiﬁcation performance.
This property is quite different from standard scalar-based architectures where each graph is modeled with only one graph embedding 2 .
By introducing the concept of capsules, the graph and class capsules can not only preserve classiﬁcation-related properties of each graph (reﬂected with the length of class capsules) but also other properties information (reﬂected with the angle of class capsules).
The generated class capsules can also be useful in other follow-up work and we leave this to be explored in the future.
C ONCLUSION
We have proposed CapsGNN, a novel framework that fuses capsules theory into GNN for more efﬁcient graph representation learning.
Inspired by CapsNet, the concepts of capsules are introduced in this architecture to extract features in the form of vectors on the basis of nodes features extracted from GNN.
As a result, one graph is represented as multiple embeddings and each embedding captures different aspects of the graph properties.
The generated graph and class capsules can preserve not only the classiﬁcation-related information but also other information with respect to graph properties which might be useful in the follow-up work and we leave this to be explored in the future.
We believe this is a novel, efﬁcient and powerful data-driven method to represent high-dimensional data such as graphs.
Our model has successfully achieved better or comparable performance when compared with other SOTA algorithms on 6 out of 10 graph classiﬁcation tasks especially on social datasets.
Compared with similar scalar-based architectures, CapsGNN is more efﬁcient in encoding features and this would be very beneﬁcial for processing large datasets.
A S PECIFIC PROCEDURE OF ROUTING
The speciﬁc procedure of routing is shown in Algorithm 1.
Algorithm of trainable transform matrices W and the number of iterations t .
1: procedure D YNAMIC R OUTING ( t, S , W ) 2: for all children capsule i : v j | i =
s Ti W ij 3: for all children capsule i to all parent capsule j : r ij ←4: for t iterations do 5: for all children capsule i :
˜ r i ← softmax ( r i ) 6: for all parent capsule j : h j ← (cid:80) i ˜
r ij v ij 7: for all parent capsule j : ˜ h
j ← squash ( h j ) 8: for all children capsule i to all parent capsule j : r ij ← r ij + ˜ h
Tj v ij 9: end for 10: return ˜ h
j 11: end procedure
B D ETAILS OF EXPERIMENTAL DATASETS
The details of benchmark datasets we use in our experiment is shown in Table 6.
[Table]
C C OORDINATE A DDITION
After Attention Module, coordinate addition can be used to preserve the position information of each node during the procedure of generating node capsule votes.
The details of Coordinate Addition module can be found in Figure 4.
This module is not necessary in some datasets.
Here, we propose this module as a selective optimization.
When the GNN goes deeper, the extracted nodes features contain more speciﬁc position information of each node.
Inspired by Zhang et al.
(2018) where the node embeddings learned from the last layer of GNN are taken to order all nodes, we also take the capsules extracted from the last layer of GNN as the position indicators of corresponding nodes by concatenating it with each capsule of the node.
The procedure of calculating votes with node position indicators can be written as:
[Equation]
where v ( n,i ) j ∈ R 1 × ( d n + d p ) represents the node capsule vote from the i th channel of the n th node to the j th channel of graph capsules.
W nij ∈ R
d × d n and W pj ∈ R d × d p are the transform matrices.
s ( n,i ) is the same as introduced in Section 3.2 and (cid:107) represents concatenate operation.
D D IFFERENCES FROM E XISTING A PPROACHES
Here, we present a general comparison between CapsGNN with existing approaches.
1) Compared with Atwood & Towsley (2016), Simonovsky & Komodakis (2017) and Zhang et al.
(2018) (GNN-based graph representation learning architectures), CapsGNN represents node features in the form of capsules.
This is helpful to preserve the properties information contained in nodes more efﬁciently when generating graph embeddings.
Besides, each graph is modeled as multiple embeddings in CapsGNN instead of only one embedding used in other approaches.
This allows us to capture information of graphs from different aspects.
The second difference is that, in these approaches, each part of the graph is given equal importance.
However, the attention mechanism used in CapsGNN allows it to assign various weights to different nodes.
This leads the model to focus on critical parts of input graphs.
Lastly, different from Atwood & Towsley (2016) and
Simonovsky & Komodakis (2017), CapsGNN and Zhang et al.
(2018) uses node features extracted from multiple layers of GNN so that different size of receptive-ﬁelds are applied to preserve more information.
2) GCAPS-CNN proposed by Verma & Zhang (2018) also introduced capsule-related concept into graph representation learning.
However, they generate capsules in a feature-based manner instead of learning capsules as distributed embeddings.
More speciﬁcally, when they extend a scalar feature to a capsule for the node n , P higher-order statistical moment value is calculated based on its neighbor nodes and these P value is concatenated to a P -dimensional capsule.
Between layers, GCAPSCNN performs dimension reduction to compress capsules back to scalar features, which defeats the purpose of having capsules in the ﬁrst place.
CapsGNN learns each dimension of capsules in a data-driven manner and apply routing mechanism between layers to preserve the learned meaning of each capsule.
This also allows us to preserve multiple properties information contained in nodes more efﬁciently especially when generating graph embeddings.
Compared with CapsNet proposed by Sabour et al.
(2017) which works well in image processing domain, CapsGNN needs to handle more complex situations when handling graphs.
In image processing domain, the size of the input images can be standardized by resizing the images.
However, it is not possible to simply resize the graphs.
So, we introduced an additional Attention Module to tackle graphs that are vastly different in sizes and preserve important parts of graphs.
We also propose to use features extracted from all layers of GNN since it is hard to deﬁne a suitable receptive-ﬁeld size for graphs.
Furthermore, compared with the architecture of CapsNet, CapsGNN has one additional graph capsules layer which is used to learn multiple graph embeddings and these embeddings reﬂect different aspects of graph properties which is valuable in future research.
To the best of our knowledge, we are the ﬁrst one to model a graph as multiple embeddings in the form of distributed capsules and we believe this approach of learning representations has a high potential for other complex data analysis which is not limited to graphs.
Besides, CapsGNN has different explanation of linear transformation.
In CapsNet, by applying a linear trainable transformation to pose vectors, the spatial relationship between object parts and the whole object can be well modeled.
However, by applying linear trainable transformation, CapsGNN is simply computing the prediction vectors from nodes-level representations to graph-level representations.
This transform matrix is not trying to model the change of viewpoint or capture viewpoint invariant knowledge but to model the relationship between the properties of nodes and the properties of the whole graph.
E E XPERIMENTAL S ETTING
The same architecture settings are used in CapsGNN for all datasets to show its robust performance. For the node capsules extraction, the GCN has 5 layers ( L = 5 ), the number of channels at each layer is set as the same which is 2 ( C l = 2 ).
The number of graph capsules is ﬁxed as 16 ( P = 16 ). The dimension of all capsules are set as 8 ( d = 8 ). The number of units in the hidden layer of Attention Module is set as 116 of the number of input units.
The number of iterations in routing is set as 3 . During training stage, we simultaneously reduce Loss c and Loss r and we scale Loss r with 0 .
1 so that the model focuses on classiﬁcation task. λ is set as 0 .
5 and {+1 . 
 for multi-class classiﬁcation+} and binary classiﬁcation respectively.
As for the node attributes in different datasets, considering that REDDIT-M5K and REDDIT-M12K are large-scale datasets with widely distributed nodes degree, we set the attributes of all nodes in these two datasets as the same which means we consider the initial node embeddings for all the nodes as the same to avoid over-ﬁtting.
For the remained relatively small datasets, both of the node degree and other node attributes are sent to CapsGNN as node features to speed up the training and we apply dropout( dropput rate is 0 . 3 ) to the input node features to improve the learning performance. The settings are summarized in the
[Table]
F C ONTRIBUTION OF E ACH M ODULE
In addition to evaluate the performance of the whole architecture on the classiﬁcation task [Table] we also provide detailed study to quantify the contributions made by each module in CapsGNN on the
1) CapsGNN (GCN + Attention + Routing + Reconstruction): Basic CapsGNN. 2) CapsGNN-Coord (GCN + Attention + Coordinate + Routing + Reconstruction): Basic CapsGNN with Coordinate Addition module. 3) CapsGNN-Avg (GCN + Average + Routing + Reconstruction): The Attention module in basic CapsGNN is replaced with the Average module.
4) CapsGNN-noRout (GCN + Attention + Reconstruction):
In this architecture, we will ﬁx the similarity coefﬁcients between all the capsules from one layer to the next layer as the same so that each children capsule will be equally routed to all the parent capsules.
5) CapsGNN-noRecon (GCN + Attention + Routing):
In this architecture, we directly remove the Reconstruction loss module.
6) CapsGNN-Avg-noRout (GCN + Routing):
In this architecture, we replace the Attention module in basic CapsGNN with the Average module and ﬁx the similarity coefﬁcients between all the capsules from one layer to the next layer as the same.
[Table]
The validation accuracy of each architecture is shown in Table 8 where we highlight the highest and lowest accuracy respectively. As we can see from the Table, social datasets(COLLAB, IMDBB) reach better performance with CapsGNN-Coord which indicates the effectiveness of Coordinate Addition Module.
However, the performance of social datasets is still comparable across all types of architectures. On the other hand, the performance of biological datasets(NCI1, PROTEINS) is more sensitive to the introduced modules in each architecture.
The highest accuracy of biological datasets is achieved on CapsGNN which indicates the little effectiveness of Coordinate Addition
Module in these two datasets. More speciﬁcally, the comparison between CapsGNN-Avg-noRout and CapsGNN-Avg on NCI1 and PROTEINS indicates the effectiveness of Routing mechanism which improves the accuracy by 2.1% and 0.6% respectively. Besides, the comparison between
CapsGNN-Avg-noRout and CapsGNN-noRout on NCI1 and PROTEINS indicates the effectiveness of Attention Module which improves the accuracy by 1.37% and 1.24% respectively. The accuracy of NCI1 and PROTEINS can be improved by as much as 2.98% and 1.33% when Attention Module and Routing mechanism are combined in the architecture.
Overall, CapsGNN is a general framework that fuses capsule theory to GNN for more efﬁcient graph representation learning.
In this framework, we also provide multiple possible modules to improve the quality of learned graph embeddings while we do not target to ﬁnd the best combination of modules for each dataset.
Since each possible module plays a different role in different datasets, it would be better to adjust the architecture and hyper-parameters based on practical situation.