B EYOND C LASSICAL D IFFUSION : B ALLISTIC G RAPH N EURAL N ETWORK
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper presents the ballistic graph neural network.
Ballistic graph neural network tackles the weight distribution from a transportation perspective and has many different properties comparing to the traditional graph neural network pipeline.
The ballistic graph neural network does not require to calculate any eigenvalue.
The ﬁlters propagate exponentially faster( σ 2 ∼ T 2 ) comparing to traditional graph neural network( σ 2 ∼ T ).
We use a perturbed coin operator to perturb and optimize the diffusion rate.
Our results show that by selecting the diffusion speed, the network can reach a similar accuracy with fewer parameters.
We also show the perturbed ﬁlters act as better representations comparing to pure ballistic ones.
We provide a new perspective of training graph neural network, by adjusting the diffusion rate, the neural network’s performance can be improved
I NTRODUCTION
How to collect the nodes’ correlation on graphs fast and precisely?
Inspired by convolutional neural networks(CNNs), graph convolutional networks(GCNs) can be applied to many graph-based structures like images, chemical molecules and learning systems.
Similar to neural networks, GCNs rely on random walk diffusion based feature engineering to extract and exploit the useful features of the input data.
Akin to neural networks(NNs), promising improvements have been achieved by deﬁning the diffusion-based ﬁlters and using them in a multi-layer NN.
Recent works show random walk based methods can represent graph-structured data on the spatial vertex domain.
For example, Li et al.
(2017) use bidirectional random walks on the graph to capture the spatial dependency and Perozzi et al.
(2014) present a scalable learning algorithm for latent representations of vertices in a network using random walks.
Except for the spatial domain, many researchers focus on approximating ﬁlters using spectral graph theory method, for example,Bruna et al.
(2013) construct a convolutional architecture based on the spectrum of the graph Laplacian;
Defferrard et al.
(2016) use high order polynomials of Laplacian matrix to learn the graphs in a NN structure model.
In Kipf & Welling (2016), the convolution is L 1 =
I + D − 1 / 2 AD − 1 / 2 .
In practice, the ﬁrst part can be seen as adding self-loops to the node and then the latter part can be seen as a random-walk based diffusion. For a k step random walk on the one-dimensional inﬁnite line with no biased walk, the ﬁnal probability distribution of the random walk will converge to applying L k on the initial state, where L = D − 1 / 2 AD − 1 / 2 .
The distance from the start point of a simple random walk will converge to C √ k , where C , k is the constant and the number of total steps respectively.
T HE SPEED PROBLEM
The random walk/Laplacian matrix is employed for collecting the correlated information over a graph.
Here we consider a two-dimensional condition, taking the start point as (0,0) and the correlated point is ( i , j ), the distance is denoted as d ij .
As discussed, the distance walker travelled is C √ k , in this case, though a (cid:100)
d ij (cid:101) step walker can reach ( i , j ), the probability distribution on the correlated point is relatively low since the walker’s average distance is C √ k .
In order to fully capture the correlation between the two vertices(in other words, increase the weight between (0,0) and ( i , j )), two main methods are used:
1. take steps (cid:100) d ij (cid:101) 2 steps, in Defferrard et al.
(2016), in analogy to a 5 × 5 ﬁlter
, the authors use Laplacian polynomial order up to 25 to achieve a similar accuracy, the number of steps is far larger than the ﬁlter’s size in CNNs.
Pooling: the distance of two vertices are shortened after pooling operation.
(cid:100)
d ij (cid:101) will reduced to (cid:100)
d ij / 2 (cid:101) / after a 2 × 2 pooling.
e g. Henaff et al.
and Bruna et al.
(2013) use max pooling, Defferrard et al.
(2016) use efﬁcient pooling and Tran et al.
2018) use sort pooling.
(cid:1) (cid:2) (cid:3)(cid:1) (cid:3)(cid:2) (cid:4)(cid:1) (cid:4)(cid:2) (cid:1)(cid:3)(cid:4)(cid:5)(cid:6) (cid:2)(cid:7) (cid:8)(cid:9)(cid:10)(cid:11) (cid:12)(cid:13)(cid:11)(cid:14) (cid:1) (cid:2)
[Figure]
(cid:6) (cid:7)(cid:8)
Classical diffusion
Figure 1 shows the relation between the distance and number of steps for the random walk/Laplacian matrix.
As the steps increases, the walker diffuse with the distance ∼ √ k , where k is the number of steps, resulting in the inefﬁciency in collecting information.
Suggested by Hammond et al.
(2011), the ﬁlter on most common graph convolutional network is:
[Equation]
= [Equation] Λ
[-(cid:80) K − 1 k =
 θ k Λ k-][Equation]
and θ ∈ R K is a vector of polynomial coefﬁcients.
The number of steps corresponds to the polynomial order of the Laplacian.
However, as the number of steps grows, the distance walker travelled changes slower and slower, resulting in low efﬁciency and duplicated ﬁlters.
In the next section, we will introduce the ballistic walk method that instead of walking at classical diffusion speed, the walker is able to reach an average distance ∼ Ck in k steps walking.
Being different from the classical diffusive transportation, this method enables us to collect correlation faster.
B ALLISTIC WALK ON GRAPH
In the following, we will focus on the regular graphs to demonstrate the ballistic graph neural network, where image, video and speech data are represented.
3. I NTRODUCTION TO B ALLISTIC WALK
The ballistic walk algorithm consists of two parts, a walker in the position space H spatial and a coin in the coin space H c .
Thus the walker is described using states in Hibert space H spatial ⊗ H c .
Let the walker initially be at the state | Ψ (cid:105)
= | i, j (cid:105) p ⊗ s 0 , where s 0 is normally a symmetric state in H c .
In analogy to the classical random walk, the next state of the walker can be expressed by | Ψ (cid:105) t +1 = (cid:98) U | Ψ (cid:105) t , where (cid:98) U consists two operations, a ﬂip operation (cid:98)
O coin in the coin space and shift operation (cid:98) S in the spatial space.
In this paper, we consider the ballistic walk on a regular two-dimensional graph.
The coin space
H c consists of four states: |↓(cid:105) , |↑(cid:105) , |←(cid:105) , |→(cid:105) , represents move up, down, left and right for the next step.
The spatial space H spatial consists N states representing the walker’s position, where N is the number of nodes.
The notation | n (cid:105) denotes an orthonormal basis for H spatial and (cid:104) n | is the
Hermitian conjugate of the state.
For a ﬁnite-dimensional vector space, the inner product (cid:104) n (cid:48) | n (cid:105) is δ nn (cid:48) and the outer product | n (cid:48) (cid:105) (cid:104) n | equals to a matrix in R N × N .
The probability stay on the node | i, j (cid:105) is (cid:80) s = ↓ , ↑ , ← , → (cid:13)(cid:13)
(cid:104) Ψ
| i, j (cid:105) ⊗ | s (cid:105) (cid:13)(cid:13) 2 .
Pseudo-code of our method is given in Algorithm 1.
Result:
The walker’s state after K steps start from ( i, j ) 1 p
= | i, j (cid:105) //
The start point 2 s 0 = a |↓(cid:105) + b |↑(cid:105) + c |←(cid:105) + d |→(cid:105) // a, b, c, d ∈ C ; (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 + (cid:107) c
(cid:107)
+ (cid:107) d (cid:107) 2 =3 (cid:98) S
= (cid:80) i,j | i − 1 , j (cid:105)(cid:104) i, j | ⊗ |↑(cid:105)(cid:104)↑| + (cid:80) i,j | i + 1 , j (cid:105)(cid:104) i, j | ⊗ |↓(cid:105)(cid:104)↓| + (cid:80) i,j | i, j + 1 (cid:105)(cid:104) i, j | ⊗ |→(cid:105)(cid:104)→| + (cid:80) i,j | i, j − 1 (cid:105)(cid:104) i, j | ⊗ |←(cid:105)(cid:104)←| 4 (cid:98)
O coin
= H ⊗ H // H is usually the Hardmard matrix
| Ψ 0 (cid:105) =
p 0 ⊗ s6 | Ψ 1 (cid:105) =
(cid:98) S ( p 0 ⊗ s 0 ) 7 for i = 2; i < K ; i = i + 1 do | Ψ
i (cid:105) =
(cid:98) S ( (cid:98)
O coin | Ψ
i − 1 (cid:105) ) 9 end
3. E XPERIMENTS
The experiment is performed on the MNIST dataset, The Hardmard matrix H is 1 √(cid:20) 1 −(cid:21) and the initial state is Ψ
= 1 j/ 2 |↑(cid:105) + 1 / 2 |↓(cid:105) − 1 j/ 2 |←(cid:105) − 1 / 2 |→(cid:105) .
Figure 2 and 3 show the difference in diffusion between the random walk based diffusion and the ballistic diffusion on a 28 × 28 grid starting from the center.
Comparing to the classical random walk, the ballistic walker shows cohesive behaviour and transports faster.
The comparison between the speed is shown in Figure 4. [Figure] [Figure]
The diffusive classical walker’s distances at time = 15 and time = 20 center around the same range.
The ballistic walkers’ differences are more signiﬁcant, which means collecting different information.
Comparing to classical random walk, the ballistic walker has a speed of ∼ k .
This linear transportation behaviour enables the ﬁlters collect correlation on graph more efﬁciently.
As shown in Figure 4, the distance for a diffusive walk at time = 25 is around taking an 8-step ballistic walk.
Defferrard et al.
(2016) considers 25 steps diffusive ﬁlters to approximate a 5 ×kernel with 10 feature maps(10 hidden units).
For comparison, we take an 8-step-ballistic kernel with the same number of feature maps.
The feature maps are then fully connected to 10/32 units and then connected to 10 units for classiﬁcation.
The notations are denoted as Ball10 and Ball32.
Tablesummarizes the capabilities of our model compared to other recent modeling approaches.
Baselines We compare our approaches with the following baselines: [Figure] �� [Figure]
� �� ��
[Figure]
Ball Ball [Figure] ���� ����� �����
[Figure]
���� ���� ���� ���� ���� ���� ����������
[Figure]
• DeepWalk : Perozzi et al.
(2014) uses local information obtained from truncated random walks.
For iterating over all the vertices of the graph, the authors generate a random walk | W v i =
t | for every node, and then use it to update representations.
• Graph Attention Networks (GAT) :
Veli ˇ ckovi ´ c et al.
(2017) assigns different weights to different nodes in a neighbourhood.
The graph attentional layer changes the weight distribution on the neighbourhood nodes.
• Manifold Regularization (ManiReg) : Belkin et al.
(2006) brings together ideas from the theory of regularization in reproducing kernel Hilbert spaces, manifold learning and spectral methods.
In the paper, their propose data-dependent geometric regularization method based on graph Laplacian.
• Graph Convolutional Network (GCN) : Kipf & Welling (2016) conducts the following layer-wise propagation in hidden layers using random walk based method( X ( k +1)
=
σ ( D − 1 / 2 AD − 1 / 2 X ( k ) W
( k ) ) ).
The ﬁnal perceptron layer for classication is deﬁnes as: Z = softmax ( D − 1 / 2 AD − 1 / 2 X ( k ) W ( k ) ) .
• Graph Learning-Convolutional Networks(GLCN) : Jiang et al.
(2019) contains one graph learning layer, several graph convolution layers and one ﬁnal perceptron layer.
The layer-wise propagation rule is: X ( k +1)
=
σ ( D − 1 / 2 s
[Equation]
( k ) ) .
R EVISIT THE SPEED PROBLEM
In the last section, we introduce the ballistic walk, which transports faster than the diffusive classical walk.
By selecting the ballistic ﬁlters up to K = 8 , we reach 97% and use 1/3 parameters comparing to spline method using classical diffusive ﬁlters.
This suggests ballistic ﬁlters are able to collect correlation more efﬁciently comparing to random walk based Laplacian ﬁlters.
Figure 6 shows the transportation behaviour of different kinds of ﬁlters.
There exists two phases: ’trapped to diffusive’ phase and ’diffusive to ballistic’ phase.
The laplacian-based ﬁlters can be seen as the up-bound ﬁlters of the trapped to diffusive phase(the orange points) and obey the √ steps law.
As the steps grow, the ﬁlters are inefﬁcient.
As shown in the Figure 6, the ﬁlters are repeatedly sampling the region with distance < 10 as the steps grow up to 70 steps, this means the ﬁltered information can be very similar, leading to invalid feature layers.
The ballistic ﬁlters lie at the up-bound of [Table] Trapped to Diffusive phase Diffusive to Ballistic phase
[Equation] [Figure] ��
[Figure] � [Equation] �� �� �� ��
[Figure]
� �� �� �� Go beyondtheboundary Localizenearthestartpoint the ’diffusive to ballistic’ phase(the blue points), the linear propagation ensures gathering the long correlation information in a relatively small number of steps.
However, linear transportation also brings drawbacks:
• Sparse sampling at the mid-distance region: as shown in Figure 6 (ﬁgure in the ﬁgure), for a 35-steps walker, the points from distance = 6 to distance = 13 enables the ballistic ﬁlters can better interpret the long correlation.
However, the distance intervals between the ballistic ﬁlters are relatively sparse, and this can result in the missing some correlation.
• Beyond the boundary: the linear ballistic transportation makes the walker go beyond the boundary (for our case the distance is 14).
With the same number of the steps, the ballistic walker travels to the boundary line(shown in Figure 7).
Is there a way to generate ﬁlters that can collect the correlation within distance < 14 area while circumventing cumbersome classical diffusion?
In other words, we are interested in generating ﬁlters with a transportation speed between ballistic and classical diffusion.
By controlling the speed of the ﬁlters, we circumvent going beyond the boundary and make all our ﬁlters localized between the regions with restricted distance(denoted as the diffusive to ballistic phase in Figure 6).
D E - COHERENCE
In the ballistic diffusion, we use Hardmard transformation on the coin space, The Hardmard operator (SU(2)) helps spilt the state in the coin space and ﬁnally leads to linear ballistic transportation.
However, as mentioned in the last section, we are interested in generating ﬁlters lines between [Figure] ballistic and classical phase so that we can circumvent the boundary and slow-transportation problem.
In this section, we introduce the de-coherence scheme to perturb ballistic transportation by adding a noise term to the Hardmard operation at every step.
This noisy perturbation results in the de-coherence of ballistic ﬁlters and thus slows down the transportation.
5. T HE INTRODUCTION TO DECOHERENCE
We want our ﬁlters have a diffusion distance in a reasonable region( a < Distance < b ).
However, the ballistic ﬁlters’ distances increase with steps.
The ﬁlters are not capable to dense sampling some speciﬁc regions.
By selecting different randomness and steps, we can generate ﬁlters localized in a bounded area.
The noisy Hardmard can be written as
H r ( β ) =
(cid:20)e iβ e − iβ −(cid:21) (2)
Randomness α =α = 0 .α
= 0 .α
= 0 .
Speed 0.
Randomness α = 0 .α
= 0 .α = 0 .α
= 0 .
Speed 0.
Table 3 shows the accuracy with different perturbated ﬁlters( α = 0 , 0 .
β
= 2 ×
R × πα denotes the randomness in the coin space, R is a random number between 0 and 1 .
The corresponding transportation speed is shown in table 2.
As the α increases to 0.20, the speed drops to 0.323.
α is a controller of the diffusion speed, as α becomes larger, the ballistic tranportation will ﬁnally evolve to the classical diffusive couterpart.
5. S UMMARY OF THE SPEED WITH RANDOMNESS
After taking randomized operations, the accuracy can be improved.
In other words, by using ﬁlters from the perturbed ballistic walk, we are now able to dense sample the ’meaningful regions’ and avoid the shallow sampling and slow transportation problem by selecting the step and the randomness of the ballistic walker.
The ’meaningful regions’ are denoted as blue and yellow in Figure 6 (ﬁgure in the ﬁgure).
In our model, we ﬁx the ﬁrst eight ﬁlters as the pure ballistic ﬁlters without perturbation.
We then select different ﬁlters from perturbed ﬁlters.
The model architecture is shown in Figure 9.
The input signals are ﬁrst passed to 25 different feature maps using the selected ﬁlters.
We then apply the convolutional operation and average pooling on the feature maps.
After a fully connected layer with 512 hidden units, the network is connected to 10 units for the classiﬁcation task.
We select our ﬁlters up to the distance<14 regions( ∼ 25 steps for ballistic duffusion), and this ensures the ﬁlters gather the correlation information within reasonable regions.
For pure ballistic ﬁlters, the classiﬁcation accuracy is around 99.11%, when we keep the ﬁrst eight ballistic ﬁlters and use different ﬁlters with different randomness, the accuracy increases to 99.39%.
Our results show that the classiﬁcation accuracy can be improved using a mixture of perturbed ﬁlters within certain distance region. [Figure] [Figure] α
[Figure]
[Figure] α [Figure] [Figure]
α
[Figure]
[Figure]
� �� �� �� ��
[Figure]
[Figure]
5. C OIN O PERATOR
The ballistic walk ﬁlters can also be generalized to different coin operators.
Except using Hardmard and noisy Hardmard coin operator, we can also use a discrete Fourier operator(DFO) or Grover operator, the discrete Fourier operator is written: DFO = d 1 /
   ...
ω ω... ω d − ... 1 ω d −ω 2( d − 1) ... ω ( d − 1)( d − 1) 
 =
1      ω ωω1 ωωω1 ωωω  (3) where ω =
e 2 πi/
d is the d th root of unity and the Grover operator is:
G =   a b b ...
b b a b ... b ... ... ... ... ... b b b ... a   = 1  −    −
[Equation]
[Equation]
[Equation]
[Equation]
(4) where a = (2 /d ) − 1 and b = 2 /d .
The results are shown in Table 4.
Note that we usually select unitary operation in the coin space to keep the probability as a constant.
However, not every unitary operator results in ballistic transportation, the Grover operator will localize near the start point as the steps grow, however, they all have a speed-up effect comparing to the classical diffusive ﬁlters.
The DFO and Grover operator have a transportation speed between the ballistic ﬁlters and diffusive ﬁlters, thus can be seen as special forms of randomized ﬁlters.
The general Hardmard coin is balanced.
DFO retains the balanced properties on a general graph.
Every state in coin space is obtained with equal probability.
The Grover operator helps retain the symmetry of the signal and is permutation symmetric.
The Grover operator is not a balanced coin because the probability(weight in our case) does not change its propagating directions( p = (1 − 2 /d ) 2 ). [Table] [Table]
C ONCLUSION AND F UTURE W ORK
In this paper [Table] we introduced a generalization of graph neural network: ballistic graph neural network.
We started from the speed problem of the traditional diffusive kernel and tackle this problem from the perspective of transportation behaviour.
We showed the linear transportation behaviour of ballistic ﬁlters and introduced the de-coherence scheme to adjust the ﬁlters’ speed.
Compared with diffusive ﬁlters, the ballistic ﬁlters achieve similar accuracy using 1/3 of the parameters.
Besides, we showed that the efﬁciency of the ballistic ﬁlters could be improved by controlling transportation behaviour.
Compared to the random walk method, we used two operators: the coin operator and shift operator to control the walker, and thus controlled the information the walker gathers.
Our pipeline provides a new perspective for efﬁcient extracting the graphical information using diffusion-related models.
Future work can investigate these two directions:
The Network Structure.
In this paper, we use simpliﬁed architecture to demonstrate the concept of the ballistic walk, the layers are limited to 5 layers, and we use traditional average pooling.
More layers can be added to improve particular accuracy, and more sophisticated pooling methods can be introducedDefferrard et al.
Other techniques like dropout can also be employed to improve accuracy.
The Ballistic Filter.
De-coherence can also be introduced into the shift operator.
In other words, we can use perturbed shifted operator, and thus we introduce randomness in the spatial domain.
We can also try different unitary operators in the coin space or change the initial state of the walker.
The extension to general graphs can be generalized by adding self-loops to the nodes and thus make the graph regular.
D ISCUSSION : B ALLISTIC F ILTER IN ONE DIMENSIONAL CONDITION
The ballistic ﬁlters are inspired by two-dimensional quantum walk.
The quantum coherence effect guarantees fast ballistic transportation.
The different states in the coin space can be seen as the independent state from spatial behaviour, for example, the spin of fermions or the polarization of light.
More information about the quantum walk can be found at Childs et al.
Why introducing ballistic ﬁlters results in better performance?
We here offer a conjecture from the perspective of signal processing using one-dimensional condition.
The classical diffusion in the one-dimensional case has the shape of:
[Equation]
[Figure]
�� � � �
[Figure]
and the frequency part can be written as:
[Equation]
The ˆ g ( f ) can be seen as a high pass ﬁlter.
For a gaussian high-pass ﬁlter, the spatial distribution is:Makandar & Halalli (2015)
[Equation]
The long time probability distribution of ballistic walk is:Luo & Xue (2015)
[Figure]
[Figure] [Equation] [Figure] Figure 11 shows the distribution of gaussian high pass ﬁlter and the cumulative distribution of 24th and 25step of ballistic diffusion.
These two distributions have a similar shape while the ballistic distribution has steeper edges resulted from fast transportation.
The ballistic ﬁlters’ capability to collect the long-time probability means it can act as a high-pass ﬁlter with different sizes.
The size of the ﬁlters depends on the walking steps.
Figure 12 shows the ballistic diffusion with a pulse signal from t = − τ to t =
τ .
The orange dashed line is an approximated shape of ballistic transportation of the leftmost signal( t = − τ ), and the blue dashed line corresponds to t = τ .
The width of the approximated shape is related to the walking steps.
For diffusive tranporation after certain steps of difussion, the region from t = − τ to t = − τ have a gaussian shape since it is sum of gaussian distribution with centers range from t
= − τ to t = − τ .
The classical diffusion acts
[Figure]