OPEN: Orthogonal Propagation with Ego-Network Modeling
Anonymous Author(s)
AfﬁliationAddress email
Abstract
To alleviate the unfavorable effect of noisy topology in Graph Neural networks(GNNs), some efforts perform the local topology reﬁnement through the pairwisepropagation weight learning and the multi-channel extension.
Unfortunately, mostof them suffer a common and fatal drawback: irrelevant propagation to one nodeand in multi-channels.
These two kinds of irrelevances make propagation weightsin multi-channels free to be determined by the labeled data, and thus the GNNsare exposed to overﬁtting.
To tackle this issue, a novel Orthogonal Propagationwith Ego-Network modeling (OPEN) is proposed by modeling relevances betweenpropagations.
Speciﬁcally, the relevance between propagations to one node ismodeled by whole ego-network modeling, while the relevance between propagations in multi-channels is modeled via diversity requirement.
By interpreting thepropagations to one node from the perspective of dimension reduction, propagation weights are inferred from principal components of the ego-network, whichare orthogonal to each other.
Theoretical analysis and experimental evaluationsreveal four attractive characteristics of OPEN as modeling high-order relationshipsbeyond pairwise one, preventing overﬁtting, robustness, and high efﬁciency.
Introduction
Graph Neural Networks (GNNs) have been proven to be a powerful tool to explore irregular graphdata by seamlessly combining graph topology and node attributes for node representation learning[1, 2].
From the perspective of spectral graph theory, GNNs, such as GCN [3] and ChebyNet [4],are proposed from graph signal ﬁltering, whose ﬁlters are derived from the topology of whole graph,and their success is attributed to low-passing ﬁltering.
From the perspective of spatial propagation,
GNNs, such as GraphSage [5] and MPNN [6], are presented by following the aggregation andcombination scheme for node attributes smoothing.
Recent progress demonstrates the equivalencebetween these two perspectives [7].
However, both the ﬁlters based on graph topology and nodeattribute propagation between neighbourhoods take the implicit assumption that the graph topologyis perfect and creditable [8].
Consequently, the oversmoothing issue [9, 10] and the expressive powerloss [11], which are caused by overusing topology via stacking multiple layers, are identiﬁed.
Actually, the graph topology exists large amount of noises.
Thus, graph topology reﬁnement iscritical to the accuracy and robustness of GNNs [12, 13].
The local reﬁnement, especially throughpairwise propagation weight learning, is widely investigated and employed due to the high accuracyand low computation load.
As a representative local reﬁnement, Graph Attention Network (GAT)[14] formulates the propagation weight on each edge as the attention between the two connectednodes, and introduces multi-channel propagation via multi-head attention to stabilize the learningprocess.
Afterward, a number of approaches are proposed by following the scheme of propagationweight learning and multi-channel extension.
For example, GaAN [15], PGCN [16] and DMP [17] extend GAT via different pairwise propagation weight learning functions.
FAGCN [18] relaxes thenon-negative constraint in GAT to a real number.
ADSF [19] extends GAT to incorporate topologystructure into the attention mechanism.
GATv2 [20] removes the ranking irrelevance limit in GAT.
Unfortunately, most GNNs, especially those with local reﬁnement, suffer a common but fataldrawback: irrelevant propagations .
Firstly, the propagations to each node are irrelevant, sincepropagation weights are either predeﬁned according to the topology or learned based on the contents ofthe two connected nodes.
Actually, the propagations to one node should reﬂect global characteristicsof its ego-network.
Secondly, propagations in multi-channels are irrelevant, since parameters tochannels are free to be learned without any speciﬁc constraints.
Actually, to obtain stable andcomplementary representations from different channels, propagation schemes in different channelsshould be diverse [17].
These two kinds of irrelevances make propagation weights free to bedetermined by the labelled data for the speciﬁc task, thus resulting GNNs being susceptible tooverﬁtting [21].
To tackle this issue, a novel Orthogonal Propagation with Ego-Network modeling (OPEN) is proposedby modeling these two kinds of relationships between propagations.
Firstly, the relevance betweenpropagations to one node is modeled by whole ego-network modeling.
Speciﬁcally, by interpretingthe propagations to one node from the perspective of dimension reduction, propagation weights areinferred from the principal component of ego-network, which corresponds to the mapping functionmaximizing the variance and captures the characteristics of the whole ego-network.
Principal component can be obtained via the eigenvalue decomposition of the covariance matrix, which is efﬁcientfor the ego-network.
Secondly, the propagations in multi-channels are implemented by employingall principal components, each of which corresponds to one channel.
Since principal componentsare orthogonal to each other, the propagations in multi-channel are diverse.
The proposed OPENpossesses some attractive characteristics:
modeling high-order relationships beyond pairwise one,robustness, and high efﬁciency.
Finally, theoretical analysis demonstrates that the two components of
OPEN, i.e., Ego-Network modeling and Orthogonal Propagation in multi-channel, can prevent theover-smoothing issue.
The main contributions of this paper are summarized as follows:
• We investigate the common but fatal issue in GNNs, i.e., irrelevant propagation.•
We propose a novel Orthogonal Propagation with Ego-Network modeling (OPEN) byinterpreting propagations in the ego-network from the perspective of dimension reduction.•
We provide a theoretical analysis of OPEN’s capability on preventing oversmoothing issue.•
We experimentally verify the effectiveness and robustness of the proposed OPEN.
Preliminaries and Analysis
Notations: Let G =
( V , E ) denote a graph with node set V
= { v 1 , v 2 , · · · , v N } and edge set E ,where N is the number of nodes.
The topology of graph G can be represented by its adjacency matrix A =
[ a ij ] ∈ { 0 , 1 } N × N , where a ij = 1 if and only if there exists an edge e ij = ( v i , v j ) betweennodes v i and v j .
The degree matrix D is a diagonal matrix with diagonal element d i = (cid:80)
Ni =1 a ijas the degree of node v i .
N ( v i ) = { v j | ( v i , v j ) ∈ E} stands for the neighbourhoods of node v i .
X ∈ R N × F and H ∈ R N × F (cid:48) denote the collection of node attribute and representation with the i throw, i.e., x i ∈ R F and h i ∈ R F (cid:48) , corresponding to node v i , where F and F (cid:48) stand for the dimensionsof attribute and representation.
Preliminaries: Most Graph Neural Networks (GNNs) follow an aggregation-combination strategy[6], where each node representation is iteratively updated by aggregating node representations ofneighbourhoods and combining the aggregated representation with the node representation itself asfollows
[Equation]
where ¯ h kv stands for the aggregated representation from neighbourhoods.
The aggregation operationis the most critical part of the message passing framework, and most GNNs utilize summarization oraverage function as the implementation of AGGREGATE k .
Therefore, most GNNs can be uniﬁed under the following formula
[Equation]
where W k is the learnable parameter and the σ ( · ) is the nonlinear mapping function.
The scalarc uv is the averaging weight, which determines the scheme of aggregation.
GNNs can be dividedinto two categories according to the design of c uv .
The methods in the ﬁrst category ﬁx c uvby regarding topology information as perfect.
For example, GCN [3] and SGC [22] set c kuv =1 / ( (cid:112) ( d u + 1)( d v + 1) , while GIN [23] sets c kuv
= 1 for u (cid:54) = v and c kvv = 1 + (cid:15) k .
The methods inthe second category tend to learn c uv by considering topology as noisy.
For example, GPRGNN [24]sets c kuv = γ k / ( (cid:112) ( d u + 1)( d v + 1) with γ k as learnable real number.
Graph Attention Network(GAT)
[14], Gated Attention Network (GaAN)
[15] and Probabilistic GCN [16] model the propagationweights as the function of the attributes of connecting nodes via normalized attention mechanism as
[Equation]
where e uv denotes the similarity between nodes v and u .
The similarity function of attributes canbe speciﬁed as e GATuv = LeakyReLU
( b k [ Wh k − 1 u ||
Wh k − 1 v ]) , e GaANuv =
( Wh k − 1 u ) T
O k
( Wh k − 1 v )and e PGCNuv = − ( Wh k − 1 u − Wh k − 1 v ) T Σ
( Wh k − 1 u − Wh k − 1 v ) where b k , O k and Σ k are learnableparameters.
Some efforts have been paid to simplify them, such as setting O k as identity matrix, i.e.,
O k
= I or constraining Σ k as diagonal matrix.
FAGCN [18] relaxes the non-negative constraint in
GAT to real number by employing tanh ( · ) instead of softmax.
GAT [14] proposes to improve the stability by employing multi-head attentions as in Transformer [25].
The multi-head attention essentially performs multi-channel propagation.
DMP [17] formates multichannel propagation as diverse message passing by enhancing Eq.
(2) via feature-wise propagationweights as
[Equation]
where (cid:12) denotes the element-wise product of vectors and the learnable propagation weight vectorc kvu has the same length as the node representation h k − 1 u .
To reduce the model complexity, DMPpresents two schemes to efﬁciently learn c kvu ’s.
Analysis: By analyzing the single-channel propagation in Eq.
(4) and the multi-channel propagationin Eq.
(4), it can be observed that existing methods possess two serious drawbacks as shown in theblue dashed box in Fig.
• Propagations to each node are irrelevant.
As shown in Eq.
(3), the propagation weight,
which is based on the contents of the two connected nodes, models pairwise relationship.
Thus, the different propagations to each node are irrelevant to each other.
However, the
node representation, which is obtained from aggregation over neighbourhood, should reﬂect global characteristics of its neighbourhood, such as high-order relationship.
• Propagations in multi-channels are irrelevant.
As shown in Eq.
(4), propagations in
different channels are free to learn without any speciﬁc constraints.
Thus, the propagation
weights in different channels may be similar and redundant.
However, to obtain stable and
complementary representations from different channels, propagation schemes in different
channels should be diverse enough.
In summary, neither the relationship between the propagations to each node nor the relationshipbetween propagations in different channels is considered.
Thus, propagation weights are free to bedetermined by the labelled and the speciﬁc task, and thus the models tend to be overﬁtting [21].
Methodology
To tackle the issues mentioned above, Orthogonal Propagation with Ego-Network modeling (OPEN)is proposed by modeling these two kinds of relationships between propagations in subsections 3.1and 3.2, respectively.
Section 3.3 provides implementation details.
Finally, section 3.4 theoreticallyanalyzes its capability on preventing over-smoothing issue.
C   x
Ego-network Extraction =
X H
=
xxx
+ +
Propagation Perspective
Dimension Reduction Perspective x =
PropagationWeight Node Embedding
MappingFunction DataPoint x
X C
L ea r n a b l e
Multi-channels
Original Graph
Pairwise Learning
Inter-channel Irrelevant
I n t r a - c h a nn e l
I rre l e va n t
Mappingvia WholeEgo-Network
O r t h ogo n a l u 1 u 2 Whole Ego-network uu
E x i s ti ng
GNN s w it h
Irr e l e v a n t
P r op a g a ti on s
P r opo s e d O P E N :
O r t hogon a l
P r op a g a ti on w it h
E go - N e t w o r k
3.1 Ego-Network Modeling
[Figure]
Firstly, the relationship between propagations to one node is modeled.
By ignoring the layer k andconsidering the ﬁrst layer, the propagation component in Eq.
(2) can be reformulated as
[Equation]
(5) where H v ∈ R |N v |× F stands for the matrix containing the representations of nodes in N v andc v ∈ R |N v | represents the propagation weights from nodes in N v to node v .
Eq.
(5) can beinterpreted from two perspectives as shown in Fig.
From the perspective of message passing, therepresentation of node v , i.e., h v is the aggregation of embeddings of its neighbourhood, which arethe rows of H v , and propagation weights often model pairwise relationship as shown in the blue solidbox in Fig.
This perspective induces the propagations to each node irrelevant as shown in the bluedashed box in Fig.
From another perspective, h v ∈ R 1 × F can be regarded as the one-dimensional representations of Fdata points, each of which corresponds to a N v -dimensional column vector of H v .
And, c v ∈ R |N v |can be seen as the mapping function, which reduces the dimension from R |N v | to R as shown in theorange solid box in Fig.
From the perspective of dimension reduction, the F representations in h vshould preserve the discriminative information in original |N v | -dimensional space of H v .
Therefore,learning the propagation weights c v in ego-network of node v is converted to the problem of seekingdimension reduction mapping function.
Among the existing dimension reduction methods, principal component analysis (PCA) is the widelyadopted unsupervised one.
The principal components correspond to the mapping functions.
Tofacilitate the description, the subscripts of H v and c v are removed, and H is represented as thecollection of column vectors as H = { h · , 1 , h · , 2 , ..., h · ,F } , each of which is an |N v | -dimensionalvector.
By denoting the mean and covariance matrix of data matrix H as ¯h = 1 F (cid:80) Fj =1 h · ,j and
S = 1 F (cid:80) Fj =
( h · ,j − ¯h )
( h · ,j − ¯h ) T , respectively, the principal components are the eigenvectors ofcovariance matrix S , i.e.
[Equation]
(6) where u j is the eigenvector corresponding to the eigenvalue λ j as shown in the orange dashed box in
Fig.
By sorting λ j in descending order, the eigenvector u 1 corresponding to largest eigenvalue λ 1 isthe ﬁrst principal component.
By using the u 1 as the mapping function c , the obtained 1-dimensionalrepresentation possesses the largest variance, and thus preserves the discriminative ability.
Ego-network of Hub Nodes: The computational complexity, which consists of construction of Sand EVD, is O
( |N v | 2 F ) .
Thus, it is efﬁcient for most nodes with small neigbourhood.
However, it may be inefﬁcient for nodes with large neighbourhood, such as hub nodes.
Thus, we present thetreatment for them.
By denoting ˜H = { h · , 1 − ¯h , h · , 2 − ¯h , ..., h · ,F − ¯h } , Eq (6) can be written as1 N ˜H ˜H T u j = λ j u j .
By multiplying both sides by ˜H T , it holds
[Equation]
By denoting z j = ˜H T u j , Eq.
(7) and u j can be respectively reformulated as
[Equation]
Since 1 N ˜H T ˜H ∈ R F × F , its construction and EVD are efﬁcient.
Thus, the complexity of ego-networkmodeling for hub nodes, which possess a large number of neighbourhoods, i.e., |N v | , is reduced from
O ( |N v | 2 F ) to O
( |N v | F 2 ) .
Therefore, ego-network for hub node can be efﬁciently modeled.
This new perspective and corresponding ego-network modeling provide following three attractivecharacteristics:
• Beyond pairwise modeling: Since the covariance matrix S ∈ R |N v |×|N v | essentially
models the correlation between nodes in the ego-network N v , its eigenvector represents
the high-order characteristics of ego-network, which is beyond the pairwise relationship
between two connected nodes.
Therefore, the propagations to each node are jointly modeled.
• Preventing overﬁtting issue: The mapping function,
i
e., the propagation weights, isinferred from the ego-network instead of learning from labelled nodes for speciﬁc task.
The
model parameters of propagation weight learning are omitted, and thus it prevent overﬁtting.
• Hight efﬁcience: The eigenvalue decomposition (EVD) is performed over the ego-networks
whose sizes are much smaller than that of whole network, and thus the EVD is efﬁcient.
3.2 Orthogonal Propagations in Multi-channels
Since multi-channel introduces multiple groups of trainable parameters, it tends to be more seriousoverﬁtting than the single-channel case.
To tackle this issue and enhance the robustness, an intuitivestrategy is to boost the diversities of different channels to provide complementary effects.
GNNsmay implement this requirement by designing diverse propagation scheme.
However, it is difﬁcult toconstrain pairwise weight learning to be diverse in an efﬁcient manner, since diversity often requiresto impose computationally expensive orthogonality constraints.
Fortunately, the employed PCA in ego-network modeling in section 3.1 simultaneously produces|N v | orthogonal components { u 1 , u 2 , ..., u |N v | } , which corresponds to different mapping functions.
Note that the PCA is efﬁcient on ego-networks with a small number of nodes.
Thus, these |N v |orthogonal components { u 1 , u 2 , ..., u |N v | } can be employed as the propagation weights in |N v |different channels to realize the diversity requirement.
By letting c jv = u j as the propagation weightsover the ego-network of node v on j th channel, Eq.
(5) can be extended to multi-channel form as
[Equation]
Discussion: Ortho-GConv [26] also introduces the orthogonality in GNNs.
However, it is verydifferent from the proposed OPEN on both motivation and methodology.
From the perspective ofmotivation, Ortho-GConv tends to alleviate oversmoothing issue, while OPEN aims at overﬁtting issue.
From the perspective of methodology, Ortho-GConv imposes orthogonality on feature transformation
W k , while OPEN presents orthogonal propagation, which is more challenging and essential in GNNs.
3.3 Model Details
Subsections 3.1 and 3.2 provide the basic ideas of ego-network modeling and extension to multichannels with orthogonal propagations.
This section presents the details on implementation.
Firstly, the number of channels is set as J in advance.
Since the number of principal components foreach ego-network is the same as the number of neighbourhood of the center node, i.e., |N| v , thesenumbers varies over nodes.
To make representations of nodes comparable, the number of channels
[Table]
is set as a hyper-parameter.
For the nodes, whose number of neighboood is less than J , the lackingcomponents are set as zero vector, i.e., c jv = 0 for j = |N| v + 1 , ..., J .
The tuning experiments on Jare shown in subsection 4.5.
Secondly, the ﬁnal representation of node can be obtained by combing representations in multichannels from Eq.
(9), such as summarization or concatenation.
To boost the expressive capability,attention mechanism is employed to combine different representations with shared learnable parameterb , i.e. [Equation]
[Equation]
Similar to existing GNNs, OPEN is trained by feeding the ﬁnal representation h v ’s into the crossentropy between predicted and ground-truth labels on labelled nodes for semi-supervised task.
Thirdly, propagation weights inferred from the original node attributes are ﬁxed for all layers.
Although it can obtain optimal solution by inferring propagation weights for the next layer from therepresentations obtained in last layer, the inference process and the back-propagation process can’tbe seamlessly combined.
For the consideration of computational complexity, propagation weightsinferred from the original node attributes are employed for all layers.
3.4 Theory Analysis on Preventing Over-smoothing
Over-smoothing issue [9, 10], i.e., representations of all nodes converge to points independent fromtheir original attributes, has been identiﬁed as the most serious issue to prevent GNNs from being deep.
The over-smoothing also leads to the exponential loss of expressive power for node classiﬁcation[11].
According to [10], over-smoothing issues is caused by that the eigenvector corresponding to thelargest eigenvalue of normalized adjacency matrix ( λ 1 = 1 ) is determined by degree, i.e., u 1 = Defor asymmetric normalization or u 1 = D 1 2 e for symmetric normalization, where e is the vector ofones.
Thus, the node representations after inﬁnite layers are
[Equation]
where w = X T u 1 is the weighted combination of attributes from all nodes, and thus it is shared byall nodes.
Therefore, the ﬁnal representation H ∞ is determined by node degree D .
To alleviate thisissue, many efforts have been paid.
Some of them modify the topology, such as DropEdge [27] and
GRAND [28], while others revise the outputs from layers, such as PairNorm [29] and Inﬂation [30].
Different from existing methods, the proposed OPEN essentially reﬁnes the topology
A according tothe attributes of nodes in ego-network X v , i.e., ˆ A = f
( A , X ) .
Therefore, it is intuitive that the ˆu 1corresponding ˆA is the composition of D and X .
According to Eq.
(11), the ﬁnal representation isrelevant to X , i.e. preventing over-smoothing issue.
The following theorem is proved in Appendix.
Theorem 1.
The representations after inﬁnite layers of OPEN are relevant to nodes attributes.
[Table]
Evaluations
[Table] Firstly, this section provides experimental setups, including dataset, baselines and implementationdetails.
Then, the node classiﬁcation results are analyzed followed by hyper-parameter tuning.
Finally,the capabilities on preventing over-smoothing and overﬁtting are veriﬁed.
4.1 Datasets
To comprehensively evaluate the proposed OPEN, 7 widely used datasets are employed.
Statistics ofdatasets are shown in Table 1.
These datasets can be divided into three categories.
• Citation Networks.
Cora, Citeseer, and Pubmed, which are widely used to verify GNNs,are standard citation network benchmark datasets [31, 32].
In these networks, nodes andedge represent papers and citations between them, respectively.
Words in the paper areemployed to represent the node feature in bag-of-word form.
The academic topic of paper istaken as the label of node.• Co-purchase Networks.
Amazon-Computers (Computers) and Amazon-Photo (Photo) aretwo networks of co-purchase relationships [33].
In these networks nodes represent goodsand edges stand for the connected two goods being frequently bought together.
Each node
owns a bag-of-words feature extracted from product reviews.
The categories of the goods are employed as the label of node. • Coauthor Networks.
Coauthor-CS (CS) and Coauthor-Physics (Physics) are two co-authornetworks based on the Microsoft Academic Graph from the KDD Cup 2016 challenge [33].In these networks nodes represent authors and edges stand for that the connected two authorshave co-authored a paper.
Each node owns a bag-of-words feature based on paper keywords.
The most active research ﬁelds of the authors are employed as the label.
4.2 Baselines
To verify the superiority of the proposed OPEN, 15 baseline methods are employed for performancecomparison.
These methods are divided into two categories.
The ﬁrst category consists of basicmethods for graph data, including the multiple layer perception (MLP), Logistic Regression (LogReg)and Label Propagation (LP) [34], Chebyshev [4], Graph Convolutional Network (GCN) [3], Graph
Attention Network (GAT) [14], GraphSAGE [5], Simple Graph Convolution (SGC) [22], MoNet [35].
The methods in the second category possess some attractive characteristics, such as preventing oversmoothing issue and high accuracy, etc This category contains Personalized Propagation of Neural
Predictions (APPNP)
[36], Jumping Knowledge Networks (JKNet) [37], GCN with Initial residualand Identity mapping (GCNII) [38], Generalized PageRank GNN (GPRGNN) [24], Diverse Message
Passing (DMP) [17] and Constrained GAT (C-GAT)
We employ the authors’ implementationsfor all baseline methods with the default hyper-parameters.
[Figure] [Figure]
4.3 Implementation Details
The proposed OPEN employs 2-layers network with K = 5 channels except for the hyper-parametertuning (Sec. 4.5) and over-smoothing investigation (Sct 4.6).
The whole network is trained in anend-to-end manner using the Adam optimizer with an initial learning rate of 0.001.
The maximumnumber of epochs is set up to 1000.
Besides, early stopping with a patience of 50 is also utilized.
Forall datasets, we randomly split nodes of each class in to 60%, 20% and 20% for training validationand testing, and run on test sets over 10 random splits, as suggested in [39].
For fair comparison, theperformance of all the methods, including baseline methods, are obtained on the same splits.
4.4 Result Analysis
The node classiﬁcation results are shown in Tab. 2.
The proposed OPEN achieves the new SOTA on6 networks in all 7 networks.
Note that the proposed OPEN not only outperforms all basic GNNs,such as GCN and GraphSAGE, but also signiﬁcantly beats almost all SOTA methods, such as GCNII,
GPRGNN and DMP.
These demonstrate the superiority of the proposed OPEN on classiﬁcationaccuracy.
This can be attributed to both the effectiveness of ego-network modeling on capturinghigh-order semantic information and the high diversity representations obtained from orthogonalpropagation in multi-channels.
Besides, the proposed OPEN also outperforms other extensions to
GAT, such as C-GAT and DMP.
This indicates that compared to other enhancements to the propagationweights learning and multi-channel learning, ego-network modeling and orthogonal propagation aremore effective.
In summary, these effectivenesses show that the modeling the irrelevance betweenpropagations is important and necessary for GNNs.
4.5 Hyper-parameter Tuning and Ablation Study
The only hyper-parameter in the proposed OPEN is the number of principal components, K , which isalso corresponds to the number of channels.
This section investigates the impact of K on classiﬁcationaccuracy.
To exclude the inﬂuences from different datasets, the impact of the number of channels in
GAT, i.e., the number of attention heads, is also investigated.
Fig.2 shows that OPEN consistentlyoutperforms GAT on all the number of channels.
However, the trends with different number ofchannels are different on different datasets.
In most cases, the superiority of OPEN is more remarkablewith large K compared to GAT.
Thus, the K is set as 5 for other experiments by balancing the accuracyand running time.
Cora
Citeseer
Pubmed
Ablation Study: The hyper-parameter tuning experiments also enable the ablation study.
When
K = 1 , only the ego-network modeling component works.
The signiﬁcant outperformance comparedto GAT demonstrates that the relevant propagation in OPEN is more superior than irrelevant propagations in GAT.
The consistent outperformances on all K ’s reveal the importance of orthogonalpropagations in multi-channels, which are also relevant.
The ablation study illustrates that bothrelevance modelings are critical and necessary.
4.6 Preventing Over-smoothing Issue
Section 3.4 theoretically analyzes the capability of OPEN on preventing oversmoothing issue.
Thissection provides experimental evaluations.
The node classiﬁcation performance changes of GCN,
GAT and OPEN with various model depths are shown in Fig.
GCN tends to be over-smoothing viaonly a few layers, since it employs topology-induced single-channel propagation.
GAT alleviates theovermoothing issue by introducing irrelevant multi-channel propagations.
Unfortunately, it also beoversmoothing after few layers.
By enhancing the GAT with relevant multi-channel propagations,
OPEN signiﬁcantly prevents the over-smoothing issue.
After many layers, the performances of OPENdo not remarkably drop.
This matches the theoretical results in Section 3.4, and veriﬁes the capabilityof relevant propagation on preventing oversmoothing issue.
4.7 Robustness
This section investigates the robustness of the proposed OPEN, which indicates whether the OPENis able to overcome the overﬁtting problem, as in [21].
To this end, experiments are conductedby randomly perturbing the edges in the testing graph.
Speciﬁcally, a set of nodes are randomlyselected according to a given sampling ratio, and then one edge is randomly added on these nodes.
Fig.
4 reports the classiﬁcation performance on testing graphs with different ratios of nodes.
Theperformances of GAT signiﬁcantly drop as the ratios of nodes increase, which indicate it tendsto be overﬁtting to the given data.
In contrary, the performance drops of the proposed OPEN arevery slight.
It indicates the robustness of OPEN.
This robustness may be attributed to two aspects.
Firstly, the propagation weights of OPEN are inferred from neighbourhoods instead of learning fromlabelled data.
Secondly, the orthogonal propagations in multi-channels promote the diversity, and thusenhance the robustness.
This investigation shows that relevant propagation modeling can promote therobustness of the GNNs.
Conclusions
This paper identiﬁes the irrelevant propagations issue in Graph Neural Networks (GNNs), whichmakes models exposed to overﬁtting to the labelled data.
The propagation irrelevance includes 1) thepropagations to one node are irrelevant, 2) propagations in multi-channels are irrelevant.
This paperpresents a novel Orthogonal Propagation with Ego-Network modeling (OPEN) to model these twokinds of relevances between propagations.
By interpreting the propagations to one node from theperspective of dimension reduction, propagation weights are inferred from the principal components ofego-network, which are orthogonal to each other.
Theoretical analysis and experimental evaluationsreveal four attractive characteristics of the proposed OPEN as modeling high-order relationshipbeyond pairwise one, preventing overﬁtting issue, robustness, and high efﬁciency.
These attractivecharacteristics reveal the importance of modeling propagation relevance. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]