OPEN: Orthogonal Propagation with Ego-Network Modeling
Abstract
To alleviate the unfavorable effect of noisy topology in Graph Neural networks (GNNs), some efforts perform the local topology reﬁnement through the pairwise propagation weight learning and the multi-channel extension.
Unfortunately, most of them suffer a common and fatal drawback: irrelevant propagation to one node and in multi-channels.
These two kinds of irrelevances make propagation weights in multi-channels free to be determined by the labeled data, and thus the GNNs are exposed to overﬁtting.
To tackle this issue, a novel Orthogonal Propagation with Ego-Network modeling (OPEN) is proposed by modeling relevances between propagations.
Speciﬁcally, the relevance between propagations to one node is modeled by whole ego-network modeling, while the relevance between propagations in multi-channels is modeled via diversity requirement.
By interpreting the propagations to one node from the perspective of dimension reduction, propagation weights are inferred from principal components of the ego-network, which are orthogonal to each other.
Theoretical analysis and experimental evaluations reveal four attractive characteristics of OPEN as modeling high-order relationships beyond pairwise one, preventing overﬁtting, robustness, and high efﬁciency.
Introduction
Graph Neural Networks (GNNs) have been proven to be a powerful tool to explore irregular graph data by seamlessly combining graph topology and node attributes for node representation learning [1, 2].
From the perspective of spectral graph theory, GNNs, such as GCN [3] and ChebyNet [4], are proposed from graph signal ﬁltering, whose ﬁlters are derived from the topology of whole graph, and their success is attributed to low-passing ﬁltering.
From the perspective of spatial propagation, GNNs, such as GraphSage [5] and MPNN [6], are presented by following the aggregation and combination scheme for node attributes smoothing [7].
Recent progress demonstrates the equivalence between these two perspectives [8].
However, both the ﬁlters based on graph topology and node attribute propagation between neighbourhoods take the implicit assumption that the graph topology is perfect and creditable [9].
Consequently, the oversmoothing issue [10, 11] and the expressive power loss [12], which are caused by overusing topology via stacking multiple layers, are identiﬁed.
Actually, the graph topology exists large amount of noises.
Thus, graph topology reﬁnement is critical to the accuracy and robustness of GNNs [13, 14, 9, 15].
The local reﬁnement, especially through pairwise propagation weight learning, is widely investigated and employed due to the high accuracy and low computation load.
As a representative local reﬁnement, Graph Attention Network (GAT) [16] formulates the propagation weight on each edge as the attention between the two connected nodes, and introduces multi-channel propagation via multi-head attention to stabilize the learning process.
Afterward, a number of approaches are proposed by following the scheme of propagation weight learning and multi-channel extension.
For example, GaAN [17], PGCN [18], Masked-GCN [19] and DMP [20] extend GAT via different pairwise propagation weight learning functions.
FAGCN [21] relaxes the non-negative constraint in GAT to a real number.
ADSF [22] extends GAT to incorporate topology structure into the attention mechanism.
GATv2 [23] removes the ranking irrelevance limit in GAT.
Unfortunately, most GNNs, especially those with local reﬁnement, suffer a common but fatal drawback: irrelevant propagations .
Firstly, the propagations to each node are irrelevant, since propagation weights are either predeﬁned according to the topology or learned based on the contents of the two connected nodes.
Actually, the propagations to one node should reﬂect global characteristics of its ego-network.
Secondly, propagations in multi-channels are irrelevant, since parameters to channels are free to be learned without any speciﬁc constraints.
Actually, to obtain stable and complementary representations from different channels, propagation schemes in different channels should be diverse [20].
These two kinds of irrelevances make propagation weights free to be determined by the labelled data for the speciﬁc task, thus resulting GNNs being susceptible to overﬁtting [24].
To tackle this issue, a novel Orthogonal Propagation with Ego-Network modeling (OPEN) is proposed by modeling these two kinds of relationships between propagations.
Firstly, the relevance between propagations to one node is modeled by whole ego-network modeling.
Speciﬁcally, by interpreting the propagations to one node from the perspective of dimension reduction, propagation weights are inferred from the principal component of ego-network, which corresponds to the mapping function maximizing the variance and captures the characteristics of the whole ego-network.
Principal component can be obtained via the eigenvalue decomposition of the covariance matrix, which is efﬁcient for the ego-network.
Secondly, the propagations in multi-channels are implemented by employing all principal components, each of which corresponds to one channel.
Since principal components are orthogonal to each other, the propagations in multi-channel are diverse.
The proposed OPEN possesses some attractive characteristics:
modeling high-order relationships beyond pairwise one, robustness, and high efﬁciency.
Finally, theoretical analysis demonstrates that the two components of
OPEN, i.e., Ego-Network modeling and Orthogonal Propagation in multi-channel, can prevent the over-smoothing issue.
The main contributions of this paper are summarized as follows:
• We investigate the common but fatal issue in GNNs, i.e., irrelevant propagation.
• We propose a novel Orthogonal Propagation with Ego-Network modeling (OPEN) by interpreting propagations in the ego-network from the perspective of dimension reduction.
• We provide a theoretical analysis of OPEN’s capability on preventing oversmoothing issue.
• We experimentally verify the effectiveness and robustness of the proposed OPEN.
Preliminaries and Analysis
Notations: Let G =
( V , E ) denote a graph with node set V
= { v 1 , v 2 , · · · , v N } and edge set E , where N is the number of nodes.
The topology of graph G can be represented by its adjacency matrix A =
[ a ij ] ∈ { 0 , 1 } N × N , where a ij = 1 if and only if there exists an edge e ij = ( v i , v j ) between nodes v i and v j .
The degree matrix D is a diagonal matrix with diagonal element d i = (cid:80)
Ni =1 a ij as the degree of node v i .
N ( v i ) = { v j | ( v i , v j ) ∈ E} stands for the neighbourhoods of node v i .
X ∈ R N × F and H ∈ R N × F (cid:48) denote the collection of node attribute and representation with the i th row, i.e., x i ∈ R F and h i ∈ R F (cid:48) , corresponding to node v i , where F and F (cid:48) stand for the dimensions of attribute and representation.
Preliminaries: Most Graph Neural Networks (GNNs) follow an aggregation-combination strategy [6], where each node representation is iteratively updated by aggregating node representations of neighbourhoods and combining the aggregated representation with the node representation itself as follows
[Equation]
where ¯ h kv stands for the aggregated representation from neighbourhoods.
The aggregation operation is the most critical part of the message passing framework, and most GNNs utilize summarization or average function as the implementation of AGGREGATE k .
Therefore, most GNNs can be uniﬁed under the following formula
[Equation]
where W k is the learnable parameter and the σ ( · ) is the nonlinear mapping function.
The scalar c uv is the averaging weight, which determines the scheme of aggregation.
GNNs can be divided into two categories according to the design of c uv .
The methods in the ﬁrst category ﬁx c uv by regarding topology information as perfect.
For example, GCN [3] and SGC [25] set c kuv = 1 / ( (cid:112) ( d u + 1)( d v + 1) , while GIN [26] sets c kuv
= 1 for u (cid:54) = v and c kvv = 1 + (cid:15) k .
The methods in the second category tend to learn c uv by considering topology as noisy.
For example, GPRGNN [27] sets c kuv = γ k / ( (cid:112) ( d u + 1)( d v + 1) with γ k as learnable real number.
Graph Attention Network (GAT) [16], Gated Attention Network (GaAN) [17] and Probabilistic GCN [18] model the propagation weights as the function of the attributes of connecting nodes via normalized attention mechanism as
[Equation]
where e uv denotes the similarity between nodes v and u .
The similarity function of attributes can be speciﬁed as e GATuv =
LeakyReLU
[Equation]
and e PGCNuv = −
[Equation]
Σ
[Equation]
) where b k , O k and Σ k are learnable parameters.
Some efforts have been paid to simplify them, such as setting O k as identity matrix, i.e.,
O k
= I or constraining Σ k as diagonal matrix.
FAGCN [21] relaxes the non-negative constraint in
GAT to real number by employing tanh ( · ) instead of softmax.
GAT [16] proposes to improve the stability by employing multi-head attentions as in Transformer [28].
The multi-head attention essentially performs multi-channel propagation.
DMP [20] formates multichannel propagation as diverse message passing by enhancing Eq.
(2) via feature-wise propagation weights as
[Equation]
where (cid:12) denotes the element-wise product of vectors and the learnable propagation weight vector c kvu has the same length as the node representation h k − 1 u .
To reduce the model complexity, DMP presents two schemes to efﬁciently learn c kvu ’s.
Analysis: By analyzing the single-channel propagation in Eq.
(4) and the multi-channel propagation in Eq.
(4), it can be observed that existing methods possess two serious drawbacks as shown in the blue dashed box in Fig.
• Propagations to each node are irrelevant.
As shown in Eq.
(3), the propagation weight, which is based on the contents of the two connected nodes, models pairwise relationship.
Thus, the different propagations to each node are irrelevant to each other.
However, the node representation, which is obtained from aggregation over neighbourhood, should reﬂect global characteristics of its neighbourhood, such as high-order relationship.
• Propagations in multi-channels are irrelevant.
As shown in Eq.
(4), propagations in different channels are free to learn without any speciﬁc constraints.
Thus, the propagation
C   x
Ego-network Extraction =
X H
=
xxx
+ +
Propagation Perspective
Dimension Reduction Perspective x =
PropagationWeight Node Embedding
MappingFunction DataPoint x
X C
L ea r n a b l e
Multi-channels
Original Graph
Pairwise Learning
Inter-channel Irrelevant
I n t r a - c h a nn e l
I rre l e va n t
Mappingvia WholeEgo-Network
O r t h ogo n a l u 1 u 2 Whole Ego-network uu
E x i s ti ng
GNN s w it h
Irr e l e v a n t
P r op a g a ti on s
P r opo s e d O P E N :
O r t hogon a l
P r op a g a ti on w it h
E go - N e t w o r k weights in different channels may be similar and redundant.
However, to obtain stable and complementary representations from different channels, propagation schemes in different channels should be diverse enough.
In summary, neither the relationship between the propagations to each node nor the relationship between propagations in different channels is considered.
Thus, propagation weights are free to be determined by the labelled and the speciﬁc task, and thus the models tend to be overﬁtting [24].
To tackle the issues mentioned above, Orthogonal Propagation with Ego-Network modeling (OPEN) is proposed by modeling these two kinds of relationships between propagations in subsections 3. and 3.2, respectively. Section 3.3 provides implementation details. Finally, section 3.4 theoretically analyzes its capability on preventing over-smoothing issue.
3.1 Ego-Network Modeling
Firstly, the relationship between propagations to one node is modeled.
By ignoring the layer k and considering the ﬁrst layer, the propagation component in Eq.
(2) can be reformulated as
[Equation]
where H v ∈ R |N v |× F stands for the matrix containing the representations of nodes in N v and c v ∈ R |N v | represents the propagation weights from nodes in N v to node v .
Eq.
(5) can be interpreted from two perspectives as shown in Fig.
From the perspective of message passing, the representation of node v , i.e., h v is the aggregation of embeddings of its neighbourhood, which are the rows of H v , and propagation weights often model pairwise relationship as shown in the blue solid box in Fig.
This perspective induces the propagations to each node irrelevant as shown in the blue dashed box in Fig.
From another perspective, h v ∈ R 1 × F can be regarded as the one-dimensional representations of F data points, each of which corresponds to a N v -dimensional column vector of H v .
And, c v ∈ R |N v | can be seen as the mapping function, which reduces the dimension from R |N v | to R as shown in the orange solid box in Fig.
From the perspective of dimension reduction, the F representations in h v should preserve the discriminative information in original |N v | -dimensional space of H v .
Therefore, learning the propagation weights c v in ego-network of node v is converted to the problem of seeking dimension reduction mapping function.
Among the existing dimension reduction methods, principal component analysis (PCA) is the widelyadopted unsupervised one.
The principal components correspond to the mapping functions.
To facilitate the description, the subscripts of H v and c v are removed, and H is represented as the collection of column vectors as H = { h · , 1 , h · , 2 , ..., h · ,F } , each of which is an |N v | -dimensional vector.
By denoting the mean and covariance matrix of data matrix H as ¯h = 1 F (cid:80) Fj =1 h · ,j and
S = 1 F (cid:80) Fj =
( h · ,j − ¯h )
( h · ,j − ¯h ) T , respectively, the principal components are the eigenvectors of covariance matrix S , i.e.
[Equation]
where u j is the eigenvector corresponding to the eigenvalue λ j as shown in the orange dashed box in
Fig.
By sorting λ j in descending order, the eigenvector u 1 corresponding to largest eigenvalue λ 1 is the ﬁrst principal component.
By using the u 1 as the mapping function c , the obtained 1-dimensional representation possesses the largest variance, and thus preserves the discriminative ability.
Ego-network of Hub Nodes: The computational complexity, which consists of construction of S and EVD, is O
( |N v | 2 F ) .
Thus, it is efﬁcient for most nodes with small neigbourhood.
However, it may be inefﬁcient for nodes with large neighbourhood, such as hub nodes.
Thus, we present the treatment for them.
By denoting ˜H = { h · , 1 − ¯h , h · , 2 − ¯h , ..., h · ,F − ¯h } , Eq (6) can be written as N ˜ H ˜ H T u j = λ j u j .
By multiplying both sides by ˜ H T , it holds
[Equation]
By denoting z j = ˜H T u j , Eq.
(7) and u j can be respectively reformulated as
[Equation]
Since 1 N ˜H T ˜H ∈ R F × F , its construction and EVD are efﬁcient.
Thus, the complexity of ego-network modeling for hub nodes, which possess a large number of neighbourhoods, i.e., |N v | , is reduced from
O ( |N v | 2 F ) to O
( |N v | F 2 ) .
Therefore, ego-network for hub node can be efﬁciently modeled.
This new perspective and corresponding ego-network modeling provide following three attractive characteristics:
• Beyond pairwise modeling: Since the covariance matrix S ∈ R |N v |×|N v | essentially models the correlation between nodes in the ego-network N v , its eigenvector represents the high-order characteristics of ego-network, which is beyond the pairwise relationship between two connected nodes.
Therefore, the propagations to each node are jointly modeled.
Preventing overﬁtting issue: The mapping function
, i
e., the propagation weights, is inferred from the ego-network instead of learning from labelled nodes for speciﬁc task.
The model parameters of propagation weight learning are omitted, and thus it prevent overﬁtting.
• Hight efﬁciency: The eigenvalue decomposition (EVD) is performed over the ego-networks whose sizes are much smaller than that of whole network, and thus the EVD is efﬁcient.
3.2 Orthogonal Propagations in Multi-channels
Since multi-channel introduces multiple groups of trainable parameters, it tends to be more serious overﬁtting than the single-channel case.
To tackle this issue and enhance the robustness, an intuitive strategy is to boost the diversities of different channels to provide complementary effects.
GNNs may implement this requirement by designing diverse propagation scheme.
However, it is difﬁcult to constrain pairwise weight learning to be diverse in an efﬁcient manner, since diversity often requires to impose computationally expensive orthogonality constraints.
Fortunately, the employed PCA in ego-network modeling in section 3.1 simultaneously produces |N v | orthogonal components { u 1 , u 2 , ..., u |N v | } , which corresponds to different mapping functions.
Note that the PCA is efﬁcient on ego-networks with a small number of nodes.
Thus, these |N v | orthogonal components { u 1 , u 2 , ..., u |N v | } can be employed as the propagation weights in |N v | different channels to realize the diversity requirement.
By letting c jv = u j as the propagation weights over the ego-network of node v on j th channel, Eq.
(5) can be extended to multi-channel form as
[Equation]
Discussion: Ortho-GConv [29] also introduces the orthogonality in GNNs.
However, it is very different from the proposed OPEN on both motivation and methodology.
From the perspective of motivation, Ortho-GConv tends to alleviate oversmoothing issue, while OPEN aims at overﬁtting issue.
From the perspective of methodology, Ortho-GConv imposes orthogonality on feature transformation
W k , while OPEN presents orthogonal propagation, which is more challenging and essential in GNNs.
3.3 Model Details
Subsections 3.1 and 3.2 provide the basic ideas of ego-network modeling and extension to multichannels with orthogonal propagations.
This section presents the details on implementation.
Firstly, the number of channels is set as J in advance.
Since the number of principal components for each ego-network is the same as the number of neighbourhood of the center node, i.e., |N| v , these numbers varies over nodes.
To make representations of nodes comparable, the number of channels is set as a hyper-parameter.
For the nodes, whose number of neighboood is less than J , the lacking components are set as zero vector, i.e., c jv = 0 for j = |N| v + 1 , ..., J .
The tuning experiments on J are shown in subsection 4.5.
Secondly, the ﬁnal representation of node can be obtained by combing representations in multichannels from Eq.
(9), such as summarization or concatenation.
To boost the expressive capability, attention mechanism is employed to combine different representations with shared learnable parameter b , i.e.
[Equation]
Similar to existing GNNs, OPEN is trained by feeding the ﬁnal representation h v ’s into the crossentropy between predicted and ground-truth labels on labelled nodes for semi-supervised task.
Thirdly, propagation weights inferred from the original node attributes are ﬁxed for all layers.
Although it can obtain optimal solution by inferring propagation weights for the next layer from the representations obtained in last layer, the inference process and the back-propagation process can’t be seamlessly combined.
For the consideration of computational complexity, propagation weights inferred from the original node attributes are employed for all layers.
3.4 Theory Analysis on Preventing Over-smoothing
Over-smoothing issue [10, 11], i.e., representations of all nodes converge to points independent from their original attributes, has been identiﬁed as the most serious issue to prevent GNNs from being deep.
The over-smoothing also leads to the exponential loss of expressive power for node classiﬁcation [12].
According to [11], over-smoothing issues is caused by that the eigenvector corresponding to the largest eigenvalue of normalized adjacency matrix ( λ 1 = 1 ) is determined by degree, i.e., u 1 = De for asymmetric normalization or u 1 = D 12 e for symmetric normalization, where e is the vector of ones.
Thus, the node representations after inﬁnite layers are
[Equation] [Equation]
where w = X T u 1 is the weighted combination of attributes from all nodes, and thus it is shared by all nodes.
Therefore, the ﬁnal representation H ∞ is determined by node degree D .
To alleviate this issue, many efforts have been paid.
Some of them modify the topology, such as DropEdge [30] and
GRAND [31], while others revise the outputs from layers, such as PairNorm [32] and Inﬂation [33].
Different from existing methods, the proposed OPEN essentially reﬁnes the topology
A according to the attributes of nodes in ego-network X v , i.e., ˆ A = f
( A , X ) .
Therefore, it is intuitive that the ˆucorresponding ˆA is the composition of D and X .
According to Eq.
(11), the ﬁnal representation is relevant to X , i.e. preventing over-smoothing issue.
The following theorem is proved in Appendix.
Theorem 1.
The representation of each node from OPEN is relevant to the principal components of its corresponding ego-network’s attribute.
[Table] [Table]
Evaluations
Firstly, this section provides experimental setups, including dataset, baselines and implementation details.
Then, the node classiﬁcation results are analyzed followed by hyper-parameter tuning.
Finally, the capabilities on preventing over-smoothing and overﬁtting are veriﬁed.
4.1 Datasets
To comprehensively evaluate the proposed OPEN, 7 widely used datasets are employed.
Statistics of datasets are shown in Table 1.
These datasets can be divided into three categories.
• Citation Networks.
Cora, Citeseer, and Pubmed, which are widely used to verify GNNs, are standard citation network benchmark datasets [34, 35].
In these networks, nodes and edge represent papers and citations between them, respectively.
Words in the paper are employed to represent the node feature in bag-of-word form.
The academic topic of paper is taken as the label of node. • Co-purchase Networks.
Amazon-Computers (Computers) and Amazon-Photo (Photo) are two networks of co-purchase relationships [36].
In these networks nodes represent goods and edges stand for the connected two goods being frequently bought together.
Each node owns a bag-of-words feature extracted from product reviews.
The categories of the goods are employed as the label of node. • Coauthor Networks.
Coauthor-CS (CS) and Coauthor-Physics (Physics) are two co-author networks based on the Microsoft Academic Graph from the KDD Cup 2016 challenge [36]. In these networks nodes represent authors and edges stand for that the connected two authors have co-authored a paper.
Each node owns a bag-of-words feature based on paper keywords.
The most active research ﬁelds of the authors are employed as the label. [Figure]
4.2 Baselines
To verify the superiority of the proposed OPEN, 15 baseline methods are employed for performance comparison.
These methods are divided into two categories.
The ﬁrst category consists of basic methods for graph data, including the multiple layer perception (MLP), Logistic Regression (LogReg) and Label Propagation (LP) [37], Chebyshev [4], Graph Convolutional Network (GCN) [3], Graph
Attention Network (GAT) [16], GraphSAGE [5], Simple Graph Convolution (SGC) [25], MoNet [38].
The methods in the second category possess some attractive characteristics, such as preventing oversmoothing issue and high accuracy, etc This category contains Personalized Propagation of Neural
Predictions (APPNP) [39], Jumping Knowledge Networks (JKNet) [40], GCN with Initial residual and Identity mapping (GCNII) [41], Generalized PageRank GNN (GPRGNN) [27], Diverse Message
Passing (DMP) [20] and Constrained GAT (C-GAT)
We employ the authors’ implementations for all baseline methods with the default hyper-parameters.
4.3 Implementation Details
The proposed OPEN employs 2-layers network with K = 5 channels except for the hyper-parameter tuning (Sec. 4.5) and over-smoothing investigation (Sct 4.6).
The whole network is trained in an end-to-end manner using the Adam optimizer with an initial learning rate of 0.001.
The maximum number of epochs is set up to 1000.
Besides, early stopping with a patience of 50 is also utilized.
For all datasets, we randomly split nodes of each class in to 60%, 20% and 20% for training validation and testing, and run on test sets over 10 random splits, as suggested in [42].
For fair comparison, the performance of all the methods, including baseline methods, are obtained on the same splits.
4.4 Result Analysis
The node classiﬁcation results are shown in Tab. 2.
The proposed OPEN achieves the new SOTA on 6 networks in all 7 networks.
Note that the proposed OPEN not only outperforms all basic GNNs, such as GCN and GraphSAGE, but also signiﬁcantly beats almost all SOTA methods, such as GCNII,
GPRGNN and DMP.
These demonstrate the superiority of the proposed OPEN on classiﬁcation accuracy.
This can be attributed to both the effectiveness of ego-network modeling on capturing high-order semantic information and the high diversity representations obtained from orthogonal propagation in multi-channels.
Besides, the proposed OPEN also outperforms other extensions to
GAT, such as C-GAT and DMP.
This indicates that compared to other enhancements to the propagation weights learning and multi-channel learning, ego-network modeling and orthogonal propagation are more effective.
In summary, these effectivenesses show that the modeling the irrelevance between propagations is important and necessary for GNNs.
4.5 Hyper-parameter Tuning and Ablation Study
The only hyper-parameter in the proposed OPEN is the number of principal components, K , which is also corresponds to the number of channels.
This section investigates the impact of K on classiﬁcation accuracy.
To exclude the inﬂuences from different datasets, the impact of the number of channels in
GAT, i.e., the number of attention heads, is also investigated.
Fig.2 shows that OPEN consistently outperforms GAT on all the number of channels.
However, the trends with different number of channels are different on different datasets.
In most cases, the superiority of OPEN is more remarkable [Figure] [Figure] with large K compared to GAT.
Thus, the K is set as 5 for other experiments by balancing the accuracy and running time.
Ablation Study: The hyper-parameter tuning experiments also enable the ablation study.
When
K = 1 , only the ego-network modeling component works.
The signiﬁcant outperformance compared to GAT demonstrates that the relevant propagation in OPEN is more superior than irrelevant propagations in GAT.
The consistent outperformances on all K ’s reveal the importance of orthogonal propagations in multi-channels, which are also relevant.
The ablation study illustrates that both relevance modelings are critical and necessary.
4.6 Preventing Over-smoothing Issue
Section 3.4 theoretically analyzes the capability of OPEN on preventing oversmoothing issue.
This section provides experimental evaluations.
The node classiﬁcation performance changes of GCN,
GAT and OPEN with various model depths are shown in Fig.
GCN tends to be over-smoothing via only a few layers, since it employs topology-induced single-channel propagation.
GAT alleviates the overmoothing issue by introducing irrelevant multi-channel propagations.
Unfortunately, it also be oversmoothing after few layers.
By enhancing the GAT with relevant multi-channel propagations,
OPEN signiﬁcantly prevents the over-smoothing issue.
After many layers, the performances of OPEN do not remarkably drop.
This matches the theoretical results in Section 3.4, and veriﬁes the capability of relevant propagation on preventing oversmoothing issue.
4.7 Robustness
This section investigates the robustness of the proposed OPEN, which indicates whether the OPEN is able to overcome the overﬁtting problem, as in [24].
To this end, experiments are conducted by randomly perturbing the edges in the testing graph.
Speciﬁcally, a set of nodes are randomly selected according to a given sampling ratio, and then one edge is randomly added on these nodes.
Fig.
4 reports the classiﬁcation performance on testing graphs with different ratios of nodes.
The performances of GAT signiﬁcantly drop as the ratios of nodes increase, which indicate it tends to be overﬁtting to the given data.
In contrary, the performance drops of the proposed OPEN are very slight.
It indicates the robustness of OPEN.
This robustness may be attributed to two aspects.
Firstly, the propagation weights of OPEN are inferred from neighbourhoods instead of learning from labelled data.
Secondly, the orthogonal propagations in multi-channels promote the diversity, and thus enhance the robustness.
This investigation shows that relevant propagation modeling can promote the robustness of the GNNs.
Conclusions
This paper identiﬁes the irrelevant propagations issue in Graph Neural Networks (GNNs), which makes models exposed to overﬁtting to the labelled data.
The propagation irrelevance includes 1) the propagations to one node are irrelevant, 2) propagations in multi-channels are irrelevant.
This paper presents a novel Orthogonal Propagation with Ego-Network modeling (OPEN) to model these two kinds of relevances between propagations.
By interpreting the propagations to one node from the perspective of dimension reduction, propagation weights are inferred from the principal components of ego-network, which are orthogonal to each other.
Theoretical analysis and experimental evaluations reveal four attractive characteristics of the proposed OPEN as modeling high-order relationship beyond pairwise one, preventing overﬁtting issue, robustness, and high efﬁciency.
These attractive characteristics reveal the importance of modeling propagation relevance. Acknowledgments and Disclosure of Funding
This work was supported in part by the National Science Fund for Distinguished Young Scholars under
Grant 62025602, in part by the National Natural Science Foundation of China under Grant 61972442,
Grant 62102413, Grant U1936210, Grants U1803263, Grant 11931015 and Grant 62276187, in part by the Key Research and Development Project of Hebei Province of China under Grant 20350802D and 20310802D; in part by the Natural Science Foundation of Hebei Province of China under
Grant F2020202040, in part by the Natural Science Foundation of Tianjin of China under Grant 20JCYBJC00650, in part by the China Postdoctoral Science Foundation under Grant 2021M703472, in part by the XPLORER PRIZE, and in part by the Fundamental Research Funds for Central
Universities.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
[Table]
A Proof of Theorem
In this section, the proof to Theorem 1 is given.
To this end, some additional notations are deﬁned in advance.
Firstly, matrix I v ∈ R N × N is deﬁned as the diagonal indicator matrix to indicate the neighbourhood of node v .
It i th diagonal element is 1, i.e., ( I v ) ii
= 1 , if and only if node i is the neighbourhood of node v , i
[Table]
e., a vi = 1 .
Thus, the relationship between the v th row of adjacency matrix A , i.e., a v , and indicator matrix I v ∈ R N × N is
[Equation]
where (cid:12) denotes the element-wise product, 1 denotes vector of ones, and I stands for identity matrix.
Thus, ˆH v = I v H only remains the rows corresponding to v ’s neighbourhoods and set other rows as zeros.
In other word, ˆH v ∈ R N × F is the padding version of H v ∈ R |N v |× F with rows of zeros.
Thus, the EVD in Eq.
(6) can be extended to
[Equation]
where ˆS = ˆH v ˆH Tv ∈ R N × N and ˆu j ∈ R N are the padding versions of S ∈ R |N v |×|N v | and u j ∈ R
|N v | , respectively.
Thus, the obtained embedding for node v from our proposed OPEN can be written as
[Equation]
According to PCA, the mapping u 1 and ˆu 1 can be expressed as the combination of the columns of
H v and ˆH v , i.e.,
[Equation]
It indicates that the propagation weights u 1 and ˆu 1 are not fully determined by the graph topology, but are mainly impacted by node attribute.
Note that different from GAT, where propagation weights are determined by attributes of connected nodes and labels, the propagation weights in OPEN are not impacted by labels any more.
Similar to Eq.
(14), the embedding obtained from asymmetric adjacency matrix, such as GraphSAGE [5], can be written as
[Equation]
Comparing Eqs.
(14) and (16), the main difference between OPEN and propagation based on asymmetric adjacency matrix is that the topology-based propagation weight 1 d v 1 in propagation based on asymmetric adjacency matrix is replaced by attribute-based propagation weight u 1 (Eq. 15) in the proposed OPEN.
Therefore, the representation of node v , i.e., h OPENv , is relevant to the principal components of node v ’s ego-network’s attribute, i.e., ˆu T 1 .
[Figure]
Citeseer
Pubmed
[Figure]
B Additional Experimental Results
This section provides additional experimental results to verify some statements and the superiority of the proposed OPEN.
B.1 Node Classiﬁcation on Another Split
Firstly, to demonstrate the superior performance of the proposed OPEN.
Node classiﬁcation results based on another split, where 20 labelled nodes per class are employed as the training as in GPRGNN [27].
The results are given in Tab. 3.
We obtain the similar conclusion as in the main body.
Especially, OPEN achieves remarkable improvements on large networks, such as Amazon-Computer and Coauthor-CS.
Since the overﬁtting issue may be more serious on large networks with little nodes labelled, this demonstratesx the capability of OPEN on preventing over-smoothing issue.
B.2 Embedding Visualization
To provide intuitive interpretation, the t-SNE visualizations of node embeddings obtained from GCN,
GAT and OPEN are given in Fig.
The regions of embeddings of nodes from different classes are with different colors.
The shapes of these regions reﬂect the characteristics of the corresponding methods.
The GCN’s regions of the embeddings for different classes are overlapped.
Thus, the GCN model tends to be under-ﬁtting.
The GAT’s and OPEN’s regions of the embeddings for different classes are distinct.
The regions of the embeddings from GAT are very sharp.
It indicates that labelled data plays a very essential rule on the embedding, which tends to be overﬁtting.
In contrary, the regions of the embeddings from OPEN are regular.
It indicates that the graph topology, which induces smoothing effect, play more important role than the labels., and thus can prevent overﬁtting.
Besides, the results from OPEN are much better than those from GAT on pubmed dataset.
GAT’s embeddings of nodes with pink color are overlapped from nodes of other colors, while OPEN’s embeddings of nodes with pink color are distinct from nodes of other colors.
This illustrates the effectiveness of the proposed OPEN
compared to GCN and GAT.
[Table]
B.3 Ablation Study on Prevent Over-smoothing Issue
Section 4.6 demonstrates the capability of OPEN on preventing over-smoothing issue.
DMP [20] proves that the diverse multi-channel propagations provide potentials to prevent over-smoothing issue
To investigate which component is more important, Ego-Network modeling or Orthogonal Propagation, ablation study is performed in this section.
To this end, the node classiﬁcation performances with different numbers of channels K and different numbers of layers are given in Fig.
It can be observed that OPEN with different K can prevent oversmoothing issue.
Thus, ego-network modeling component, which is equivalent to OPEN with K = 1 , can prevent over-smoothing issue.
Note that it is hard to only employ orthogonal propagation without ego-network modeling.
Thus, we do not experimentally evaluate the effectiveness of this component.
DMP [20] shows that the diversity of the channels promote the ability on preventing over-smoothing issue.
Essentially, the orthogonality in multi-channels in OPEN realize the requirement on diversity.
Therefore, both two components can prevent over-smoothing issue
[Table]
B.4 Running Time Comparisons
To demonstrate the efﬁciency of the proposed OPEN, the running time comparisons are given in
Table.
OPEN-Weight and OPEN-Propagation are the weight calculation and propagation parts of the OPEN.
Compared to OPEN, where weights are ﬁxed, OPEN-layer denote the variant which learns weights in each layer.
The results show that OPEN has the similar running time as GAT.
The running time of GAT and OPEN is similar.
Note that the running time of GAT and OPEN is longer than that of GCN, due to their multiple-channel propagations and combinations.
Besides, while the weight calculation is efﬁcient compared to propagation, the afford of layer-wise weight calculation, i.e., OPEN-layer, is too high.
These meet our complexity analysis in Section 3.1.
C Algorithm Description of OPEN
To make the OPEN easy to follow, algorithms description are given.
Table.
5 provides the egonetwork modeling algorithm, whose output is the propagation weights.
Table.
6 is the algorithm of
OPEN-layer, which performs the ego-network modeling in each layer.
Table.
7 is the ﬁnal OPEN algorithm, where the ego-network modeling is only performed once.
do do % Ego-network modeling %
Generate ego-network of v from { H ( l − 1) , A } : H ( l − 1) v ;
% Generate propagation weights for l-th
Generate propagation weights of v : { u ( l −
1 ,v u via Algorithm
1 on l
; % Orthogonal propagations in
-channels %
Update representation for for
7: OPEN
Algorithm
OPEN
Input : Feature matrix X ∈ R N × F , Adjacency matrix A ∈ R N × N
Output : Node representations: H for v =1 to N do % Ego-network modeling %
Generate ego-network of v from { X , A } : H v ; % Generate propagation weights for all layers %
Generate propagation weights of v : { u 1 ,v , u 2 ,v , . . .
, u J,v } via Algorithm-1 on H v ; end for for l =1 to L do for v =1 to N do % Ego-network modeling %
Generate ego-network of v from { H ( l − 1) , A } : H ( l − 1) v ;
% Orthogonal propagations in multi-channels %
Update representation of v of channel j: h ( l ) j,v T = u
Tj,v H ( l − 1) v , j = 1 , 2 , . . .
, J ;
Update representation of v from different channels via Eq . 10; end for end for return H ( L )
= { h ( L ) 1 , h ( L ) 2 , . . .
, h ( L ) N } .