I TERATED GRAPH NEURAL NETWORK SYSTEM
Anonymous authors
Paper under double-blind review
A BSTRACT
We present Iterated Graph Neural Network System (IGNNS), a new framework of Graph Neural Networks (GNNs), which can deal with undirected graph and directed graph in a uniﬁed way.
The core component of IGNNS is the Iterated
Function System (IFS), which is an important research ﬁeld in fractal geometry.
The key idea of IGNNS is to use a pair of afﬁne transformations to characterize the process of message passing between graph nodes and assign an adjoint probability vector to them to form an IFS layer with probability.
After embedding in the latent space, the node features are sent to IFS layer for iterating, and then obtain the high-level representation of graph nodes.
We also analyze the geometric properties of IGNNS from the perspective of dynamical system.
We prove that if the IFS induced by IGNNS is contractive, then the fractal representation of graph nodes converges to the fractal set of IFS in Hausdorff distance and the ergodic representation of that converges to a constant matrix in Frobenius norm.
We have carried out a series of semi supervised node classiﬁcation experiments on citation network datasets such as citeser, Cora and PubMed.
The experimental results show that the performance of our method is obviously better than the related methods.
I NTRODUCTION
GNN (Scarselli et al., 2009) has been proved to be effective in processing graph structured data, and has been widely used in natural language processing, computer vision, data mining, social network and biochemistry.
In recent years, GNN has developed a variety of architectures, such as GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2018), DGI (Veli ˇ ckovi ´
c et al., 2019), GIN (Xu et al., 2019), GCNII (Ming Chen et al., 2020) and GEN (Li et al., 2020).
These architectures have a common feature, that is, the representation of each node is updated using messages from its neighbors but without distinguishing the direction (or angle) of message passing between two nodes.
Recent studies have shown that considering directed message passing between nodes can improve the performance of GNN and achieve success in related ﬁelds.
For example, DimeNet (Klicpera et al., 2020) considers the spatial direction from one atom to another and can learn both molecular properties and atomic forces.
R-GCN (Schlichtkrull et al., 2018) and
Bi-GCN (Marcheggiani & Titov, 2017; Fu et al., 2019) are models for directed graph, applied in the ﬁeld of natural language processing.
We note that the above direction based model does not consider the bidirectional mixed passing of messages.
But in real life, message passing is interactive in different directions.
For example, node A obtains a message from node B.
After processing the message, node A not only passes it to the next node
C, but also feeds back to node B. Suppose there are only two directions for message passing, forward and backward, represented by 0 or 1, respectively.
The symbol space of the ﬁrst generation message passing path is { 0 , 1 } = { 0 , 1 } 1 , and that of the second generation message passing path is { 00 , 01 , 10 , 11 } = { 0 , 1 } 2 .
Generally, the symbol space of the n -th generation message passing path is { 0 , 1 } n and the size of the symbol space is 2 n .
This means that the scope of message passing spreads with exponent 2.
However, in Bi-GCN (similar to Bi-LSTM) and R-GCN architectures, the symbol space is {{ 0 } n , { 1 } n } , and its size is 2, which indicates that a lot of information will be lost in the process of message passing (see Appendix A).
How to characterize the above message passing patterns?
We use two mappings to represent message passing process in two directions.
Then the interactive passing of messages in different directions is equivalent to the composite operation of corresponding mappings.
In addition, the direction of message passing is often random, so we endow the two mappings with an adjoint probability vector to reﬂect the randomness.
Because the symbol space of the iterative path of the Iterated Function
System (IFS) with two mappings is also { 0 , 1 } n and the mapping is selected with a certain probability, the iterative process of IFS is similar to the message passing process.
In other words, the above message passing pattern can be described perfectly by IFS with probabilities.
We naturally present the Iterative Graph Neural Network System (IGNNS), whose core layer is constructed by IFS.
Figure 1 describes the differences in message passing patterns among GCN, Bi-GCN and
IGNNS.
At the same time, we regard undirected graph as a directed graph with equal probability of bidirectional message passing (see Figure 1(a)), so the IGNNS architecture can handle directed graph and undirected graph in a uniﬁed way.
P RELIMINARIES
A graph G = ( V, E ) is deﬁned by its note set V = { v 1 , v 2 , ..., v N } and edge set E = { ( v i , v j ) | v i , v j ∈ V } .
Let A ∈ R N denote the adjacency matrix of G , providing with relational information between nodes.
A [ i, j ] denote i, j th element of A ,
A
[ i, :] means the i th row, and
A [: , j ] means the j th column.
In this paper, we assume that all nodes of G are self adjacent, that is A [ i, i ]
= 1 , i = 1 , 2 , ..., N . let D = diag
( d 1 , d 2 , ..., d N ) be the degree matrix of A , where d i = (cid:80) Nj =
1 A [ i, j ] .
Neighborhood Normalization.
There are two ways to normalize A .
One approach is the following mean-pooling employed by Hamilton et al.
and Veli ˇ ckovi ´ c et al.
(2019) for inductive learning:
[Equation]
Another approach is the following symmetric normalization employed by Kipf & Welling (2017):
[Equation]
Iterated Function System.
A mapping f : R N → R N is said to be a contractive mapping on R N if there exists a constant 0 < c < 1 such that (cid:107) f
( x 1 ) − f ( x 2 ) (cid:107) 2 < c (cid:107) x 1 − x 2 (cid:107) 2 for all x 1 , x 2 ∈ R N .
An iterated function system (Hutchinson, 1981) is deﬁned by
[Equation]
where each f i : R N → R N is a contractive mapping and p
= ( p 1 , p 2 , ..., p n ) is an adjoint probability vector meaning that f i is selected by probability
p i for each iteration.
Hutchinson (1981) showed that there exists a unique nonempty compact set F such that F = (cid:83) ni
=
f i ( F ) .
We call F the fractal set or invariant set of IFS.
More conclusions on IFS can be found in the Appendix C. It is well known that there exists a unique probability measure µ with support F satisfying the equation
[Equation]
  1 1 01 1 00 1 10 1 1  , A 0 =                   
                   .
The probability measure µ in (1) is called the self-similar measure of IFS with probability vector p .
IGNNS A RCHITECTURE
In this section, we will introduce the architecture of the IGNNS according to the input layer, IFS layer, representation layer and output layer, which is described in Figure 2.
3. I NPUT L AYER
Given a graph structure data X ∈ R N × F of G =
( V, E )
, called as the feature matrix of node set V .
A row of X represents the F -dimensional feature vector of a node in V .
Let W int ∈ R F
×
H be a learnable parameter matrix, where H is the dimension of the latent space.
Then XW int ∈ R N × H .
The output of input layer is deﬁned by
[Equation]
(2) where σ ( · ) is the activation function.
Generally, ReLU ( x ) = max(0 , x ) is used as the nonlinear activation function.
Here, each column of X int is regarded as a point in R N , so X int is the set of H points in R N and arranged in a certain order.
The vector composed of the i th component of these points (the i th row of X int ) is a feature representation of the i th node of graph G .
3. IFS L AYER
Let A be the adjacency matrix of G .
Let triu ( A ) denote the upper triangular matrix of A and tril ( A ) denote the lower triangular matrix of A .
The symmetric normalization of triu ( A ) and tril ( A ) are
[Equation]
and A 1 = D − 12 1 tril ( A
[Equation]
D −
[Equation]
where D 0 and D 1 are degree matrices of triu ( A ) and tril ( A ) respectively.
Sometimes, we use the mean-pooling of triu ( A ) and tril ( A ) , i.e. A
= D −
0 triu ( A ) , A 1 = D − 1 1 tril ( A ) .
Let f 0 , f 1 be the two afﬁne transformations on R N , induced by A 0 , A 1 respectively, deﬁned as follows:
[Equation]
where b 0 and b 1 are learnable biases, namely add constants b 0 and b 1 to each component of A 0 x and
A 1 x respectively.
Constructing iterated function system
[Equation]
where p =
( p 0 , p 1 ) is a learnable adjoint probability vector, satisfying p 0 > 0 , p 1 > 0 and p 0 + p 1 = 1 .
Using the symbol space Ω m = { 0 , 1 } m , then for each i
= ( i 1 , i 2 , ..., i m )
∈
Ω m the length of i is m , denoted as | i | = m , and deﬁning p i
= p
i 1 p i 2 · · · p i m and f i = f i 1 ◦ f i 2 ◦ · · · ◦ f i m .
Let n be the number of iterations of IFS.
For IGNNS, n is a preset parameter.
The iterative process of IFS is described as follows:
The ﬁrst iteration ( | i | = 1 ).
The result of the ﬁrst iteration is denoted by
[Equation]
where H i =
f i ( X int ) , ∀
i ∈
Ω 1 .
Since IFS selects the iteration branch f
i with probability p
i , the mathematical expectation of H (1) is computed by
[Equation]
If choose to use bias in iterations, then H
= A 0 X int + b 0 , H 1 =
A 1 X int + b 1 , where b 0 and bare learnable H -dimensional vectors.
The second iteration ( | i | = 2 ).
Using the results of the ﬁrst iteration as the input of the second iteration, then the result of the second iteration is denoted by
H
[Equation]
) =
{ f 0 ( f 0 ( X int )) , f 0 ( f 1 ( X int )) , f 1 ( f 0 ( X int )) , f 1 ( f 1 ( X int )) }
= { f 00 ( X int ) , f 01 ( X int ) , f 10 ( X int ) , f 11 ( X int ) }
=
{ H i } | i | =2 ,
where H i = f i ( X int ) , ∀ i ∈
Ω 2 .
Note that IFS selects the iteration path
f i with probability p
i , then the mathematical expectation of H (2) is computed by
[Equation]
We expand the expression of E 2 and perceive its powerful feature representation ability.
First,
H
[Equation]
=
[Equation]
int + b 0 ) + b 0 , H
= f
( H 1 )
=
A 0 ( A 1 X int + b 1 ) + b 0 ,
H 10 = f 1 ( H 0 ) =
A 1 ( A 0 X int + b 0 )
+ b 1 , H 11 = f
( H 1 )
=
A 1 ( A 1 X int + b 1 ) + b 1 .
Then
E= p
H
+ p 01 H
+ p 10 H 10 + p 11 H= ( p
A
+ p
A 01 + p 10 A 10 + p 11 A 11 ) X int + ( p
A 0 b 0 + p
A 0 b 1 + p 10 A 1 b 0 + p 11 A 1 b 1 )
+ ( p 00 b 0 + p 01 b 0 + p 10 b 1 + p 11 b 1 ) , where A i =
A
i 1 A
i 2 , ∀ i
=
( i 1 , i 2 ) ∈
Ω 2 .
The n -th iteration ( | i | = n ).
Inductively, we have
[Equation]
Note that
H i ∈
R N × H and each column of H
i is a point in R N , so we regard it as a subset of R N with H elements.
Thus H ( n ) is a subset of R N with H × 2 n elements (including duplicate elements).
Because of Theorem 4.1, we call H ( n ) the fractal representation with depth n of notes.
3. R EPRESENTATION L AYER
After n iterations of IFS layer
, the dynamic trajectory of IFS is obtained:
[Equation]
In general, the global representation R of notes is obtained by time average or concatenation operations on O .
[Equation]
where (cid:107) is the concatenation operator.
Because of Theorem 4.2, we call R the ergodic representation of notes.
In practice, we adopt weighted time average or weighted concatenation.
According to the
Theorem B.1, we use heuristic weights (Here, we understand it as the average expansion factor of the distance between two points after afﬁne transformation).
Let r = (cid:112) ln
( N )
+ γ , where γ ≈ 0 .
577215664 is the Euler constant.
Suppose r = ( r 1 , r 2 , ..., r n ) is a learnable n-dimensional vector with initial value
r i = (cid:0) 1 r (cid:1) i − 1 .
Then the ergodic representation of notes is
[Equation]
3. O UTPUT L AYER
Let W out ∈ R H
× P be a learnable parameter matrix, where P is the dimension of the output layer (such as the number of class labels).
If R is generated by O concatenation, then let W out ∈ R nH × P .
There are two ways to construct output layer, one is to use a multi-layer perception as output, that is
O =
MLP ( R , W out ) ; the other is to use f 0 , f 1 for mixed propagation , that is, let R 0 = f
( RW out ) and R 1 = f 1 ( RW out ) , where the biases of f 0 , f 1 are removed, then the output
O =
[Equation]
, where the bias b out ∈ R P is an optional learnable parameter vector.
3. I NITIALIZATION OF L EARNABLE V ARIABLES
The learnable parameters of IGNNS include input layer matrix W int ∈ R F × H , adjoint probability vector p
= ( p 0 , p 1 ) ∈
R 2 of IFS, biases b 0 , b 1 ∈ R H of IFS layer, weight coefﬁcient r = ( r 1 , r 2 , ..., r n ) of representation layer, matrix W out ∈ R H
× P of output layer and bias b out ∈ R P of output layer.
Among them, W int and W out are the required learnable parameters, using the initialization described in Glorot & Bengio (2010); b 0 , b 1 and b out are optional learnable parameters with an initial value 0 ; p is a optional learnable parameter, for undirected graph, setting p 0 ∈ [0 .
5 − 0 .
5 + 0 .
1] , and for directed graph, setting (For the reasons, see Appendix F)
[Equation]
r is a optional learnable parameter, and its initial value as deﬁned in 3.3.
Let n be the number of iterations of IFS.
We regard n as the depth of IGNNS.
Thus, IGNNS is denoted as
O = IGNNS ( X , A ; W int , n, p , b 0 , b 1 , r , W out , b out ) or simply O = IGNNS
( X , IFS ) , where IFS is induced by relational matrix A .
The output of IGNNS can be used as the input of downstream tasks, and can also be connected to other network architectures.
3. T HEORETICAL T IME C OMPLEXITY OF IGNNS
Let n, N, H, P be deﬁned as above.
Let COUNT ( · ) denote the number of calculations of an object.
For input layer,
COUNT ( input layer ) =
NFH + NH = O ( NFH ) .
For IFS layer, during the iterative calculation, we will store the previous calculation results, thus
COUNT ( H (1) ) =
2 N 2 H , COUNT ( H ( i ) )
= 2 × COUNT ( H ( i − 1) ) , i = 2 , 3 , ..., n .
It follows that
COUNT ( H ( i ) )
=
2 i N 2 H, i = 1 , 2 , ..., n .
Similarly, COUNT ( { p i
|| i
| = i } )
=
2 i, i = 1 , 2 , ..., n .
Complete the above calculation, it is easy to see that COUNT ( E i )
=
2 i NH .
Thus
COUNT ( IFS layer ) =
[Equation]
i
[Equation]
It is easy to verify that COUNT ( representation layer )
=
O ( nNH ) .
We assume that the output layer is constructed by mixed propagation, then COUNT ( output layer )
=
O ( N 2 P ) if W out ∈ R H × P , and COUNT ( output layer )
=
O ( N 2 P + nNHP ) if W out ∈ R nH × P .
Then
COUNT ( IGNNS ) =
O (2 n N 2 H + N 2 P ) or O (2 n N 2 H + N 2 P + nNHP ) .
In practice, for large graphs, 2 n N 2 H (cid:29) N 2 P (cid:29) nNHP , thus 2 n N {+
+} H is the main factor affecting the time complexity of IGNNS.
Furthermore, for large graphs of the same size, n is the main important factor affecting time complexity.
For citation network datasets such as citeser, Cora and PubMed, we suggest that n ≤ 8 .
G EOMETRIC P ROPERTIES OF IGNNS
The discussion here assumes that afﬁne f 0 , f 1 are contractive.
Otherwise, let f 0 : x →(cid:107)
A 0 (cid:107) F +1 A 0 x + b 0 and f 1 : x → 1 (cid:107) A 1 (cid:107) F +1 A 1 x + b 1 .
In practice, IGNNS does not use contractive afﬁne.
If contractive afﬁne is used in IGNNS, it can be seen from the following theorems that the characterization ability of IGNNS decreases with the increase of IFS iterations, which is similar to the performance of Graph Convolution Network (GCN).
Such a phenomenon is called over-smoothing (Li et al., 2018b; Xu et al., 2019;
Chen et al., 2020), which suggests that as the number of layers increases, the representations of the nodes in GCN are inclined to converge to a certain value and thus become indistinguishable.
In order to overcome the over-smoothing problem of deep GNN, people need to use some new methods such as Skip connection (Xu et al., 2018), Drop edge (Rong et al., 2020), Residual connection (Klicpera et al., 2019a), Identity mapping (Chen et al., 2020), Generalized message aggregation functions (Li et al., 2020) and so on.
Generally speaking, deep network may lead to the decrease of generalization performance.
To analyze which type of deep GNN would achieve better generalization performance, Xu et al.
(2020) proposes a guiding theoretical framework.
Theorem 4.1 (Fractal generation) Let H ( n ) =
{ H i } | i | = n , which is a subset of R N with H × 2 n elements (including duplicate elements), then
[Equation]
where d H is the Hausdorff distance deﬁned on H ( R N ) , the set of all nonempty compact subsets of R N , and F is the fractal set of IFS in IGNNS.
In other words, as the number of iterations increases, H ( n ) will be independent of node feature X , only related to the graph structure described by A .
Let T be the Hutchinson operator on H ( R N ) , deﬁned as T ( B ) = f
( B ) (cid:83) f 1 ( B ) , ∀ B ∈ H ( R N ) .
Then the updated rule of H ( n ) satisﬁes
[Equation]
where H (0) = { X int } is a subset of R N with H elements.
In fractal geometry, H ( n ) is used to draw the fractal image on the plane.
First, taking initial value H (0) =
{ x 0 } , where x 0 is a point in plane.
For enough n , printing all the points of H ( n ) on the screen to obtain the approximate fractal image.
Theorem 4.2 (Ergodic property) Let E n = (cid:80) | i | = n p i H i be the mathematical expectation of
H ( n ) =
{ H i } | i | = n , then E n converges to a constant matrix E ∈
R N × H in Frobenius norm, i.e. lim n →∞ E n = E , where E [ i,
:] =
( e i , e i , ..., e i ) (cid:62) ∈ R H and e i ∈ R is a constant, i = 1 , 2 , ..., N .
Furthermore, the time average of the dynamic trajectory O of IFS satisﬁes
[Equation]
and series (cid:80) ∞
i =1 r i E i ∈
R N × H converges in Frobenius norm.
[Table]
Theorem 4.2 shows that as long as the number of iterations is large enough, the embeddings of nodes will be close to linear correlation, and the representation ability of IGNNS will decline.
However, in the framework of IGNNS, because the spectral radius ρ ( A 0 ) =
ρ ( A 1 ) = 1 , IFS is not contractive in general, and IGNNS still has the ability of depth feature representation.
E XPERIMENTS 5. E XPERIMENTAL T ASK : S EMI - SUPERVISED N ODE C LASSIFICATION
Let Z = softmax ( O ) , where Z ∈ R N and softmax ( · ) is the softmax activation function, deﬁned as softmax ( x i )
= 1 Z exp( x i ) with Z = (cid:80) i exp( x i ) , is applied row-wise.
For semi-supervised multiclass classiﬁcation, we employ the following cross-entropy to evaluate error over all labeled examples:
L = − (cid:80) l ∈ Y L (cid:80) Pi =1 Y [ l, i ] ln Z [ l, i ] , where Y L is the set of node indices that have labels with P classes, Y [ l, :] is a one-hot vector of size P representing the class of node l and Z [ l, :] is the row l of the matrix Z .
5. E XPERIMENTAL S ETUP
Dataset.
In our experiment, we use three standard citation network benchmark datasets for evaluation, including Cora, Citeseer, Pubmed and apply the standard ﬁxed training/validation/testing split (Yang et al., 2016; Kipf & Welling, 2017;
Veli ˇ ckovi ´ c et al., 2018) on above datasets, withnodes per class for training, 500 nodes for validation and 1,000 nodes for testing.
In these citation networks, papers are represented as nodes, and citations of one paper by another are denoted as edges.
Node features are the bag-of-words vector of papers, and node label is the only one academic topic of a paper.
See Table 1 for more details.
Parameter Setting.
Random seed for Tensorﬂow and Numpy is set to 1234.
ReLU (Nair & Hinton, 2010) is used as the activation function in input layer and output layer.
Dropout (Srivastava et al., 2014) is applied to input layer, IFS layer and output layer.
In representation layer, we adopt weighted time average mean to get the global representation of notes.
In output layer, we adopt the method of mixed propagation to get the output of IGNNS.
We use the AdamOptimizer (P.Kingma & Ba, 2015) during training.
More details of hyper-parameters are shown in Table 2.
During training stage, we select the best model to maximize the accuracy of the validation set and use early stopping with a patience of 100 epochs.
5. E XPERIMENTAL R ESULT
We compare with those models that strictly follow the standard of experiment setup of semisupervised node classiﬁcation, i.e. the standard ﬁxed training/validation/testing split (Yang et al., 2016; Kipf & Welling, 2017) is applied on dataset.
For baselines, we include recent deep GNN models such as JKNet (Xu et al., 2018), APPNP (Klicpera et al., 2019a), Attention-based models such as GAT (Veliˇckovi´c et al., 2018), AGNN (Thekumparampil et al., 2018) and H-GAT (Gulcehre et al., 2019) , and other models such as TAGCN (Du et al., 2017) and N-GCN (Abu-El-Haija et al., 2018).
We also include three state-of-the-art shallow GNN models: Planetoid (Yang et al., 2016),
GCN (Kipf & Welling, 2017) and DGCN (Zhuang & Ma, 2018).
The detailed results are shown in
Table 3.
We can see from Table 3 that the improved performance of model IGNNS in dataset Cora and
Citeseer is much higher than that in dataset Pubmed.
To understand why this happens, we analyze
[Table] [Table]
et al., 2019)
[Table]
the characteristics of these citation networks.
We consider two statistical properties of networks, one is
the Network Density d ( G ) , which is deﬁned as d ( G ) = 2 L N ( N − 1) , where N is the number of nodes and L is the number of edges, and the other is the Average Clustering Coefﬁcient C , which is deﬁned as C = 1 N (cid:80) i ∈
V C
i , where V is the set of nodes, C i = 2 e i k i ( k i − 1) , k i is the number of the neighbors of node v i and e i is the number of undirected edges between k i neighbors.
The less Network Density means the more global sparsity of the network, and the less Average Clustering Coefﬁcient means the more sparsity of the neighbors of nodes.
The calculation results of the statistical characteristics of the network are shown in Table 4.
We can see from Table 4 that Pubmed is more sparse than
Cora and Citeseer.
The performance of IGNNS beneﬁts from the bidirectional mixed propagation of information between nodes, but this sparsity weakens the gain of IGNNS.
5. P ERFORMANCE OF COMPLETELY LINEAR IGNNS
In Nonlinear IGNNS, we use the nonlinear activation function ReLU ( x ) , learn adjoint probability vector p
= ( p 0 , p 1 ) by p 0 ←
ReLU ( p 0 )
+0 .
1 ReLU ( p 0 )+ ReLU ( p 1 )+0 . 2 , p 1 ←
ReLU ( p 1 )+0 .
1 ReLU ( p 0 )+ ReLU ( p 1 )+0 . 2 and learn the representation layer coefﬁcient r = ( r 1 , r 2 , ..., r n ) by r
i ← ReLU ( r i ) with initial value r i = (cid:0) 1 r (cid:1) i −where r
= (cid:112) ln( N )
+ 0 .
In this experiment, to get a completely linear IGNNS, we let all the activation functions be the identity function, i.e. σ ( x ) = x , and let the adjoint probability vector p
=
( p 0 , p 1 ) and the representation layer coefﬁcient r = ( r 1 , r 2 , ..., r n ) be hyperparameters
[Table] [Table]
without learning.
For Citeseer, we let p
= 0 .
6 , and for Pubmed, we use bias for IFS layer.
Except for the above changes, experimental task and the other experimental setups remain unchanged as showed in section 5.1 and 5.2 respectively.
We can see from Table 5 that the performance of completely linear IGNNS is better than that of baseline model GCN.
Compared with other models, completely linear IGNNS is still competitive.
This is due to the fact that the IFS can extract more features than spectral ﬁlters.
For more discussion, Let A be the normalization adjacency matrix of graph G , and let A
and A 1 be deﬁned as in section 3.2.
We further assume that the dimension of the hidden space is equal to 1.
This means that the input of GNN (GCN or IGNNS) is a point x 0 =
X int =
XW int ∈ R N × 1 .
Let n be the depth of GNN, for IGNNS, equal to the number of iterations of IFS.
For convenience, we ignore the activation function and parameter matrix.
Let f ( x ) =
A x
+ b , f 0 ( x )
=
A 0 x + b 0 and f 1 ( x ) =
A 1 x + b 1 .
For GCN, the message passing results are
[Equation]
First, we see that each iteration only gets one value, i.e. |{ f n ( x 0 ) }| = 1 .
If f is a contractive mapping, then by Banach ﬁxed point theorem we have f n ( x 0 )
→ x fixd ∈ R N .
This means that when n is large, the features become indistinguishable.
If f is not a contractive mapping, we consider the higher order part of f n ( x 0 ) , i.e. A n x 0 .
Then A n x 0 → x 0 or A n x 0 → D − 12 x 0 ; see Xu et al.
for more details.
This also means that when n is large, the features become indistinguishable.
For IGNNS, the message passing results are as follows:
[Equation]
If f 0 and f 1 satisfy separation condition, i.e. f 0 ( x ) = f 1 ( y ) implies x = y , then |{ f i ( x 0 ) } | i | =
n | = 2 n .
This means that IGNNS can extract more information than GCN.
Even if f 0 and f 1 are contractive mappings, by Theorem 4.1, we have { f i ( x 0 ) } | i | = n → F , where F is the fractal set of IFS induced by f 0 and f 1 .
Generally speaking, F is a uncountable compact set.
This means that when n is large, the features are still distinguishable.
C ONCLUSION
In this paper, we propose a universal framework of graph neural networks, IGNNS, which give a connection between Graph Neural Networks and Iterated Function System.
We use IFS to simulate the bidirectional message passing process of graph neural network, and obtain the fractal representation and ergodic representation of graph nodes, which are very helpful for downstream tasks.
The experiments show that we have achieved good results in semi-supervised node classiﬁcation task.
Interesting directions for future work include pruning the iterative path space { 0 , 1 } n to reduce the computational complexity, coding graph structured data with IFS, and establishing more interesting connections between IFS and graph neural networks.
A T HE FRACTAL REPRESENTATION OF GRAPH G WITH ONLY ONE SELF ADJACENT NODE v .
For the sake of discussion we assume that the dimension of the hidden space is equal to 1.
We assume that messages are sent from node v , propagate in two directions (clockwise and anticlockwise) and are ﬁnally received by node v .
The received messages in the clockwise direction become one-third of the original and the received messages in the anticlockwise direction become one-third of the original plus a constant of 23 .
It is expressed by mathematical formula as follows
[Equation]
For Bi-GCN, messages are delivered independently in both directions.
In other words, there are two independent channels and the message passing (transmitting or receiving) can only be carried out by their own channels.
Let x 0 ∈ R be the initial message.
In the clockwise direction, after n passes, the received message is
[Equation]
In the anticlockwise direction, the received message is
[Equation]
For IGNNS, the two channels have a connection point at node v .
ﬁrst, node v sends the message xin both directions, and the connection point of node v will receive two message { f 0 ( x 0 ) , f 1 ( x 0 ) } .
In the second launch, any message ( f 0 ( x 0 ) or f 1 ( x 0 )) can be sent in both directions, so the received message is { f 0 ( f 0 ( x 0 )
[Equation]
f
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
f
[Equation]
x
[Equation]
In summary, After n passes, the received message is
[Equation]
where C is the famous Cantor Set.
This means that we have not only received one message, but 2 n , since f 0 and f 1 satisfy the separation condition.
We can see from Figure 3 that Bi-GCN gets boundary messages and IGNNS gets all messages.
Let p =
( p 0 , p 1 ) be the adjoint probability vector, then the mathematical expectation E n of H ( n ) is (cid:80) | i | = n p i f i ( x 0 ) .
We interpret E n as the average of all messages received.
Now the question is, fractal representation gets enough messages, but is there redundancy in these messages?
How to select the valid message from the fractal representation becomes the focus of our research in the next stage.
B F ROBENIUS N ORM OF M ATRIX
Theorem B.1 Let A ∈ R N × N be the adjacency matrix of a graph with no weights, i.e. A i,j =
if there exists an edge i → j in the graph and A i,j = 0 otherwise, and
A
=
D − 1 1 tril ( A ) (or
A 1 = D − 12 1 tril ( A ) D − 12 1 ) as deﬁned in IGNNS, then
[Equation]
where γ ≈ 0 .
577215664 is the Euler constant.
Proof.
Case 1: A 1 = D − 1 1 tril ( A ) .
Let tril ( A ) =
( a ij ) N × N , D 1 = diag
( d 1 , d 2 , ..., d N ) be the degree matrix of tril ( A ) and A 1 = ( b ij ) N × N .
Note that tril ( A ) is an lower triangular matrix, then d i =
[Equation]
Nj =1 a ij = (cid:80) ij =1 a ij .
Since a ij ∈ { 0 , 1 } , we have i ≥ d i .
Computing the Frobenius Norm of A 1 as follows:
[Equation]
∀ i ∈ { 1 , 2 , ..., N } , note that d i elements in { a ij } ij =1 are 1 and the rest are 0.
It follows that
[Equation]
It follows from (4) and (5) that
[Equation]
So (cid:107)
A
[Equation]
F
[Equation]
=1 1 i ≈
[Equation]
ln( N ) + γ , where γ ≈ 0 .
577215664 is the Euler constant.
Case 2: A 1 = D − 12 1 tril ( A ) D − 12 1 .
Computing the Frobenius Norm of A 1 as follows:
[Equation]
∀ i ∈ { 1 , 2 , ..., N } , it follows from j ≥ d j that
[Equation]
Note that d i elements in { a ij } ij =1 are 1 and the rest are 0.
It follows from Rearrangement inequality that
[Equation]
where a i ( i − d i +1)
= a i ( i − d i +2)
= · · · = a ii = 1 .
Thus
[Equation]
It follows from (6), (7) and (9) that
[Equation]
Which completes the proof.
C I NTRODUCTION TO I TERATED F UNCTION S YSTEM
In order to prove Theorem 4.1 and Theorem 4.2, we will brieﬂy introduce the relevant conclusions on IFS in this section, and we will not give the proof here.
More details of IFS Theory can be found in Hutchinson (1981);
Elton (1987); Barnsley (1988); Falconer (1990); Massopust (2017).
C. F RACTAL S PACE
Let ( X ; d ) be a complete metric space.
Let H ( X ) denote a set consisting of all nonempty compact subsets of X .
Hausdorff distanceon d H on H ( X ) deﬁned by
[Equation]
Theorem C.1 ( H ( X ); d H ) is a complete metric space.
We call ( H ( X ); d H ) a Fractal space.
Let { f i } ni =1 be a set of mappings on ( X ; d ) .
Hutchinson operator T : ( H ( X ); d H ) →
( H ( X ); d H ) deﬁned as
[Equation]
Theorem C.
If { f i } ni =1 is a set of contractive mappings on ( X ; d ) , then Hutchinson operator T is a contractive mapping on ( H ( X ); d H ) .
C.
M ARKOV O PERATOR OF IFS
Let ( X ; d ) be a complete metric space.
Let M ( X ) be the set of all probability measures on X .
Let C ( X ) be the set of all continuous functions mapping X to R .
We say that f ∈ (cid:93) Lip 1 , if | f ( x ) − f ( y ) | ≤
d
x, y ) , ∀ x, y ∈ X .
It is easy to see that if f ∈ (cid:93) Lip 1 then f ∈ C ( X ) .
Hutchinson metric d M on M deﬁned as
[Equation]
Theorem C.3 ( M ( X ); d M ) is a complete metric space.
Let IFS =
{ X ; f 1 , f 2 , ..., f n ; p } , the Markov operator M : M ( X ) → M ( X ) of IFS deﬁned as
[Equation]
Theorem C.4 Markov operator M of IFS is a contractive mapping on space ( M ( X ); d M ) .
Let measure sequence { µ i } ⊂ M and µ ∈ M , we call { µ i } weakly convergent to µ if the following equation holds:
[Equation]
[Equation]
denoted as µ i w −→ µ as i → ∞ .
Theorem C.
If µ i d M −−→ µ as i → ∞ , then µ i w −→ µ as i → ∞ .
D T HE P ROOF OF T HEOREM 4.
Proof.
By Theorem C.1, ( H ( R N ); d H ) is a complete metric space.
Let T be the Hutchinson operator on H ( R N ) , deﬁned as T ( B ) = f
( B ) (cid:83) f 1 ( B ) , ∀ B ∈ H ( R N ) .
By Theorem C.2, T is a contractive mapping on ( H ( R N ); d H ) .
It follows from the Banach ﬁxed point theorem that there exits a unique compact subset F of H ( R N ) such that
[Equation]
which implies that F is the fractal set of IFS.
Further more, ∀ B ∈ H ( R N ) , we have T n ( B ) d H −−→ F .
The above convergence is independent of the choice of initial value.
Thus, let H (0) =
{ X int } = { x 1 , x 2 , ..., x H } , x i ∈ R N , we have
[Equation]
The above result indicates that when n is large enough, H ( n ) is close to the fractal set F of IFS in the sense of Hausdorff distance, and has nothing to do with the choice of initial value H (0) .
E T HE P ROOF OF T HEOREM 4.
Proof.
It sufﬁces to prove that ∀ j ∈ { 1 , 2 , ..., H } , E n [: , j ] , the j column of E n , satisﬁes
[Equation]
For this purpose, let x j be the j column of X int as deﬁned in (2), then x j is a point in R N .
Deﬁne a
Dirac measure as follows:
[Equation]
It is easy to see that δ x ∈ M ( R N ) .
The Markov operator M : M ( R N ) → M ( R N ) of IFS, deﬁned as
[Equation]
Now take µ
= δ x j , and the results of iterative calculation are as follows:
[Equation]
Inductively, we have
[Equation]
By Theorem C.4, it follows from the Banach ﬁxed point theorem that there exits a unique probability measure µ ∗ such that
[Equation]
The above µ ∗ is actually the self-similar measure of IFS.
By Theorem C.5, we have µ n w −→ µ ∗ , i.e.
[Equation]
It follows from (17) and (3) that
[Equation]
In (18)
[Equation]
∀ i ∈ { 1 , 2 , ..., N } , take the continuous function F i to satisfy
[Equation]
It follows from (18) that ∀ i ∈ { 1 ,
[Equation]
[Equation] [Equation]
Which combined with simple mathematical analysis technology completes the proof
[Equation]
F H OW TO S ET THE I NITIAL V ALUE OF A DJOINT P ROBABILITY V ECTOR ?
The geometric meaning of matrix determinant det A is the expansion factor of graph volume under linear transformation A .
Let F be the fractal set of IFS as deﬁned in IGNNS.
Then
[Equation]
Note that
[Equation]
It can be seen that if the value of det A i is large, it reﬂects that f i ( F ) has a large share in F .
therefore, when selecting the iterative function, f i should have a greater probability of being selected.
So set
[Equation]
Note that A 0 , A 1 are triangular matrixes and the diagonals of triu ( A ) , tril ( A ) are equal to 1.
It follows from A
= D − 1 0 tril ( A ) (or A
= D − 12 0 tril ( A ) D − 12 0 ) that det A 0 = 1 det D 0 .
Similarly, det A 1 = 1 det D 1 .
It follows that
[Equation]
where D 0 and D 1 are degree matrices of triu ( A ) and tril ( A ) respectively.