Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks
Equal Contribution
Abstract
The core operation of current Graph Neural Networks (GNNs) is the aggregation enabled by the graph Laplacian or message passing, which filters the neighborhood node information.
Though effective for various tasks, in this paper, we show that they are potentially a problematic factor underlying all GNN methods for learning on certain datasets, as they force the node representations similar, making the nodes gradually lose their identity and become indistinguishable.
Hence, we augment the aggregation operations with their dual, i.e., diversification operators that make the node more distinct and preserve the identity.
Such augmentation replaces the aggregation with a two-channel filtering process that, in theory, is beneficial for enriching the node representations.
In practice, the proposed two-channel filters can be easily patched on existing GNN methods with diverse training strategies, including spectral and spatial (message passing) methods.
In the experiments, we observe desired characteristics of the models and significant performance boost upon the baselines on 9 node classification tasks.
Introduction
As a generic data structure, graph is capable of modeling complex relations among objects in many real-world problems [32, 41, 13].
Motivated by the success of Convolutional Neural Networks (CNNs) [31] on images, graph convolution [54] is defined on the graph Fourier domain and the node spatial neighborhood domain [58], respectively, in the form of spectral- and spatial-based methods.
Based on the 2 methodologies, different (linear) graph filters and (non-linear) deep learning techniques [30] are combined, giving rise to Graph Neural Networks (GNNs), achieving remarkable progress [6, 22, 19, 28, 52, 38, 37].
Most existing graph filters can be viewed as operators that aggregate node information from its direct neighbors.
Different graph filters yield different spectral GNNs or spatial aggregation functions.
Among them, the most commonly used is the renormalized affinity matrix [28].
By adding an identity matrix to the adjacency matrix, i.e., a self-loop in the graph topology, renormalized affinity matrix is created as a low-pass (LP) filter [39] mainly capturing low-frequency signals, which are locally smooth features across the whole graph [53].
Aggregation processes, in the form of message passing used in spatial-based methods, as in e.g., GraphSAGE [22] and GraphSAINT [57], are also node-level
LP filters which make nodes become similar to their neighbors.
The main idea of neighborhood feature aggregation is to exploit the intrinsic geometry of the data distribution: if two data points are close (or connected) to each other on the manifold, they should also be close to each other in the representation space.
This assumption is usually referred to as manifold (local invariance) [1, 24, 8, 7], assortative mixing (assortativity) [42], homophily [40, 36, 33] or smoothness [27, 35] assumption, which plays an essential role in the development of various kinds of algorithms including dimensionality reduction [1] and semi-supervised learning [61] 2 .
This assumption naturally holds in many real world networks [40, 26], e.g., social networks, citation networks, evolutionary biology etc. .
However, in contrast to homophily, there also exists a large number of heterophily networks where individuals with diverse characteristics tend to gather in the same group [45], e.g., dating networks [63] and fraudsters in online purchasing networks [43].
On these networks, there is no strong reason to impose smoothness assumption and on the contrary, non-smoothness pattern between nodes turns out to be important.
With the above in mind, in this paper we first propose a method to measure the smoothness of the input features and output labels of an attributed graph based on Dirichlet energy and graph signal energy.
With the proposed method, we measure the smoothness of 9 real world datasets, which shows that signal defined on graph is generally a mixture of smooth and non-smooth graph signals and each part plays an indispensable role.
Motivated by this discovery, we argue that, to learn richer representations, the distinctive information between nodes should also be extracted.
Hence, we design a two-channel filterbank (FB) [14] GNN framework which use low-pass (LP) and high-pass (HP) filters together to learn the smooth and non-smooth components, respectively.
FB-GNN framework can easily be plugged into spatial methods, with LP filter for aggregation operation and HP filter for diversification operation.
With experiments on 9 real world datasets, we find that the the HP channel indeed plays and important role in the representation learning, and one-channel baseline methods can gain significant performance boost after being augmented by two-channel methods.
Preliminaries
After introducing the prerequisites, in this section, we formalize the idea behind graph signal filtering.
We use bold fonts for vectors ( e.g., v ), vector blocks ( e.g., V ) and matrix blocks ( e.g., V i ).
Suppose we have an undirected connected graph G =
( V , E , A ) without bipartite component, where V is the node set with |V|
= N , E is the edge set, A ∈ R N × N is a symmetric adjacency matrix with A ij =if and only if e ij ∈ E otherwise A ij = 0 , D is the diagonal degree matrix, i.e., D ii = (cid:80)
j A ij and
N i = { j : e ij ∈ E} is the neighborhood set of node i .
A graph signal is a vector x ∈ R N defined on
V , where x i is defined on the node i .
We also have a feature matrix X ∈ R N × F whose columns are graph signals and each node i has a feature vector X i, : with dimension F , which is the i -th row of X .
2.1 Graph Laplacian and Affinity Matrix
The (Combinatorial) graph Laplacian is defined as L = D − A , which is a Symmetric Positive
Semi-Definite (SPSD) matrix[11].
Its eigendecomposition gives L = U Λ U T , where the columns of
U ∈ R N × N are orthonormal eigenvectors, namely the graph Fourier basis , Λ = diag( λ 1 , . .
. , λ N ) with λ 1 ≤ · · · ≤ λ N and these eigenvalues are also called frequencies .
The graph Fourier transform of the graph signal x is defined as x F = U − 1 x =
U T x = [ u T 1 x , . .
, u TN x ] T , where u T i x is the component of x in the direction of u i .
A smaller λ i indicates a smoother basis function u i defined on G [12], which means any two elements of u i corresponding to two connected nodes will have more similar values.
This is because finding the eigenvalues and eigenvectors of graph Laplacian is actually solving a series of conditioned minimization problems relevant to the smoothness of the function defined on G .
Some variants of graph Laplacians are commonly used in practice, e.g., the symmetric normalized
Laplacian L sym =
D − 1 / 2 LD − 1 / 2 =
I − D − 1 / 2 AD − 1 / 2 , the random walk normalized Laplacian
L rw =
D − 1 L = I − D − 1 A .
L rw and L sym share the same eigenvalues, which are inside [0 , 2) , and their corresponding eigenvectors satisfy
u i rw
=
D − 1 / 2 u i sym .
The affinity (transition) matrix derived from L rw is defined as A rw =
I − L
rw =
D − 1 A and its eigenvalues λ
i ( A rw ) = 1 − λ i ( L rw ) ∈ ( − 1 , 1] .
Similarly, A sym =
I − L sym
= D − 1 / 2 AD − 1 /is an affinity matrix as well.
Renormalized affinity matrix is introduced in [28] and defined as
ˆ A rw =
˜ D − 1 ˜ A , where ˜
A ≡ A + I, ˜ D ≡ D
+ I and λ ( ˆ A rw ) ∈
( − 1 , 1] .
It essentially defines a random walk matrix on G with a self-loop added to each node in V and is widely used in GCN as follows,
[Equation]
where W 0 ∈ R F × F 1 and W 1 ∈ R F 1 × O are parameter matrices.
ˆ L rw can be defined as I − ˆ A rw .
ˆ A sym ≡ ˜ D − 1 / 2 ˜
A ˜ D − 1 / 2 can also be applied in GCN and ˆ L sym
= I − ˆ A sym .
Specifically, the nature of transition matrix makes ˆ
A rw behave as a mean aggregator ( ˆ A rw x ) i =
(cid:80) j ∈{N i ∪ i } x
[Equation]
/ ( D
[Equation]
+ 1) which is applied in [22] and is important to bridge the gap between spatial- and spectral-based graph convolution methods.
2.2 Measure of Smoothness and (Dirichlet) Energy
Dirichlet Energy is often used to measure how variable a function is [17] and for signal defined on graph, it can measure the global smoothness of the signal [48, 5, 49] and is defined as follows.
Definition 1.
(Dirichlet Energy)
The Dirichlet energy of vector block X and column vector x defined on G are separately defined as
[Equation]
Note that E G S is always non-negative since L is SPSD.
The graph signal energy is defined as follows.
Definition 2.
(Graph Signal Energy [18, 51])
The signal energy of block vector X and column vector x defined on undirected graph G are separately defined as
[Equation]
The signal energy represents the amount of contents in a graph signal and we will draw the correlation between E G S and E G and explain how they can be used to measure the smoothness and non-smoothness of a graph (block) signal.
Take column vector x for example, E G S ( x ) can be written as,
[Equation]
The frequency λ i before (cid:13)(cid:13)
u Ti x (cid:13)(cid:13) 2 2 can be considered as a scalar weight and E G S ( x ) focuses on measuring the component of x in the direction of non-smooth u i , who has a large weight
λ
i .
A small
E G S ( x ) means x does not contain much non-smooth components.
E G ( x ) can be written as
[Equation]
Signal x can be decomposed into smooth and non-smooth components, and the amount the nonsmooth component can be measured by
[Equation]
Note that E G NS ( x ) can be negative and a small E G NS ( x ) indicates that x is highly non-smooth.
Upon the above analysis, we define S ( x ) to measure the smoothness of a signal as follows
[Equation]
Graph signal with a small S -value means it is a smooth function define on G .
S can be different depends on the Laplacian we use to train GNN and S can be larger than 1.
In this paper, we use L sym and ˆ L sym to measure the smoothness
Filterbanks in GNNs: From One Channel to Two
In this section, we state why it is necessary to switch to the two-channel filtering process from only one-channel.
Then, we propose the filterbank-GNN framework which can learn a mixture of smooth and non-smooth graph signals
[Table]
3.1 Motivation
[Table]
We measure the smoothness of 9 frequently used benchmark datasets and present the results with the network characteristics for each task in Table 1).
It shows that the input features and ground truth labels of different datasets are all mixtures of smooth and non-smooth graph signals but in different proportions.
Besides, it illustrates that different tasks have different demands of learning how to smoothen the input signals.
For example, in Cora , Citeseer and Pubmed , the ground truth labels are much smoother than the input features, such pattern motivates us to learn how to smoothen the input signals; while in Wisconsin and Texas , the labels are less smooth than the input features, thus there is no reason that we still learn how to smoothen the input signals.
Therefore, to accommodate different situations, we propose that we should learn both smooth and non-smooth components of the input features adaptively instead of merely extracting the smooth part.
This motivates us to use filterbanks (LP and HP filters) to filter the signals in GNNs.
LP, HP Graph Filters and Filter Banks
The multiplication of L and x acts as a filtering operation over x , adjusting the scale of the components of x in frequency domain.
To see this, consider
[Equation]
The projection u i u Ti x corresponding to a large | λ i | will be amplified, while the one corresponding to a small | λ i | will be suppressed.
More specifically, a graph filter that filters out smooth (non-smooth) components is called HP (LP) filter.
Generally, the Laplacian matrices ( L sym , L rw , ˆ L sym , ˆ L rw ) can be regarded as HP filters [14] and affinity matrices
( A sym , A rw , ˆ A sym , ˆ A rw ) can be treated as LP filters [39].
In general, we denote HP and LP filters as L HP and L LP respectively.
On the node level, left multiplying HP and LP filters on x can be understood as diversification and aggregation operations, respectively.
For example, if we implement L rw and A rw on the i -th node, we have
[Equation]
Intuitively, HP filters depict the differences between one node and its neighbors; While LP filters focus on the similarity within a neighborhood, from which we can obtain missing or “hidden” features of one node.
We believe that these two conjugate components are both indispensable to portray a node.
Mathematically, multiplying with LP filter (aggregation) is a linear projection, which will project the features to a fixed subspace.
We will lose the expressive power by only using LP filter, and the missing half is the HP component of the learned signals, as L LP + L HP
=
I , which satisfies the perfect reconstruction property [14].
The two-channel linear filterbank which contains a set of filters L LP and L HP is widely used in graph signal processing [15, 14], but are rarely used in graph neural networks.
Inspired by this technique, we propose the two-channel filterbank GNNs in section 3.2, which can extract both smooth and non-smooth components from input features.
3.2 Filter Bank assisted GNNs (FB-GNNs)
X
O
X H
X L
ReLU( ⋅ ; W H )
ReLU( ⋅ ; W L )
HP
LP α L α H
A
ReLU( ⋅ )
Spectral-based FB-GNNs We use previously defined L LP and L HP (see section 3.1) to construct the two-channel FB-GNNs as follows (learning framework is provided in figure 1)
[Equation]
where H 0 =
X ; W l − 1 L , W l − 1 H ∈ R F l − 1 × F l are learnable parameter matrices for the non-linear feature extractor focusing on disentangling the smooth and non-smooth information from input H l − 1 , separately; α lL , α lH ∈ [0 , 1] are learnable scalar parameters which can learn the relative importance of H lL and H lH and keep a balance between them; l is the layer number and suppose the FB-GNN has n layers.
In this way, the hidden output H l is able to learn a mixture of smooth and non-smooth signals.
Spatial-based FB-GNNs Inspired by (6) and (7), the two-channel spatial-based method can be implemented by designing different aggregator (LP filter) and diversification operator (HP filter) as follows,
[Equation]
[Equation]
where W l − 1 L , W l − 1 H ∈ R F l × F l − 1 are learnable parameter matrices to extract LP and HP features for two channels; w ij is the connection weight between node i and node j derived from adjacency matrix, it can be a fixed value or a learnable attention coefficient such as [52]; α lL , α lH ∈ [0 , 1] are learnable scalar parameters; l is the layer number.
Computational Cost: Parameters and Runtime
The spectral two-channel learning introduces additionally one GCN operation and one weighted sum (with negligible costs introduced with nonlinearity and weighted sum before output); For spatial methods, similarly, the two-channel learning introduces one additional node-wise subtraction and one additional weighted sum for training on each pair of nodes.
Thus, the computational cost and the number of parameters are approximately doubled;
For runtime, overlooking the minor overhead of synchronization, the computations introduced by the additional pass are naturally parallelizable with the original pass (for their independently associated parameters) both in the forward and backward passes.
Therefore, no significant additional computational time will be incurred on modern GPU architectures.
Related Works
Dirichlet energy Dirichlet energy (more generally in p -Dirichlet form) is usually used as a regularizer or objective function to impose local neighborhood smoothness in various machine learning tasks, e.g., spectral clustering [1], image processing [16, 4, 60], non-negative matrix factorization [7], matrix completion, principal component analysis (PCA), semi-supervised learning [64, 65, 2].
It has different names in different literature, e.g., Laplacian regularizer [60], manifold regularizer [7], quadratic energy function [64], etc. .
The definition of smoothness derived from Dirichlet energy in (4) can be considered as a continuous relaxed form of normalized cut (ratio cut) [20, 50], which is closely related to graph partition problems.
Instead of using Dirichlet energy as a part of loss function during training process, we point out that combining with graph signal energy, it can be used to measure the smoothness of the input features and output labels for a given learning task.
With this, the necessity of learning the non-smoothness component can be confirmed.
Measuring Smoothness
The authors of [44] propose a node homophily to measure the smoothness of ground truth labels of dataset as follows,
[Equation]
[63] proposes edge homophily ratio, which is the fraction of edges that the connected nodes share the same label ( i.e., , intra-class edges).
Both of these methods do not provide an extension definition on block vector.
Thus, they fail to measure the smoothness of the input features and cannot be used to compare the difference of smoothness between the input features and labels.
[59] proposes row-diff and col-diff to measure the average of all pairwise distances between the node features and the average of pairwise distances between columns of the representation matrix.
But quantifying pairwise distance is inconsistent with the definition of smoothness introduced in section 1 which focuses on measuring the distance between connected nodes.
[37] studies homophily from post-aggregation node similarity perspective.
[35] uses statistical hypothesis testing to detect the effect the edge bias.
On Addressing Heterophily Geom-GCN [44] uses a geometric aggregation scheme and a bi-level aggregator to capture the information of structural neighborhoods, which can be distant nodes.
These can efficiently take use of the geometric relationships defined in the latent space.
H 2 GCN [63] designs ego- and neighbor-embedding separation, aggregation of higher-order neighborhoods, and combination of intermediate representations to generalize the limitation of existing GNNs beyond homophily setting.
Non-local GNNs [34] propose a simple and effective non-local aggregation framework with an efficient attention-guided sorting for GNNs.
CPGNN [62] models label correlations through a compatibility matrix, which is beneficial for heterophilic graphs, and propagates a prior belief estimation into the GNN by using the compatibility matrix.
FAGCN [3] learns edge-level aggregation weights as GAT [52] but allows the weights to be negative, which enables the network to capture high-frequency components in the graph signals.
GPRGNN [10] uses learnable weights that can be both positive and negative for feature propagation.
This allows GPRGNN to adapt to heterophilic graphs and to handle both high- and low-frequency parts of the graph signals.
BernNet [23] designs a scheme to learn arbitrary graph spectral filters with Bernstein polynomial to address heterophily.
The aforementioned works design various tricks, trying to take use of multi-hop neighborhood information and capture long-range dependencies with the belief that heterophily problem could be alleviated with the help of the distant nodes.
Although these methods show some promising results, the effectiveness is limited and do not jump out of the scope of neighborhood aggregation.
In this paper, We target directly its cause, handling heterophily problem by seeking the distinctiveness between nodes with an additional channel to learn the non-smoothness components.
Experiments
In this section, we first validate whether the two-channel filtering and learning procedure lead to better representation learning when patched on popular shallow GNN baselines 3 : GraphSAINT [57],
GraphSAGE [22], Graph Attention Network (GAT) [52], GCN [28], Geom-GCN-P (-S and -I) [44] and Graph Wavelet Neural Network (GWNN) [55].
Deeper GNNs are shown to have the potentials of mitigating the heterophily problem by extracting multi-hop neighborhood information.
For them, we test the two-channel framework on two state-of-the-art methods GCNII and GCNII* [9], with varied model depths.
After these, we validate the effectiveness of each proposed component with a detailed ablation test.
The experiments are conducted in the form of node classification 4 under supervised learning setting and performed on 9 datasets including Cornell , Wisconsin , Texas , Actor , Chameleon , Squirrel , Cora ,
Citeseer , and Pubmed (details to be found in the appendix).
Their rough characteristics are shown in
Table 1.
5.1 Experimental Setup
In supervised learning of shallow GNNs, we keep the same training configurations for GraphSAINT and FB-GraphSAINT on the 9 datasets, which are the random walk sampler with length 2 (RW) setting 5 in GraphSAINT [57]; for GWNN [55] and FB-GWNN, we use the same hyperparameters s = 1 . 0 , t = 10 − 4 on the 9 datasets 6 .
Other GNNs and their two-channel variants are under the same experiment settings as [22] and [44].
We use ˆ A sym as low-pass spectral filter and ˆ L sym as high-pass spectral filter 7 .
For supervised learning on deep GNNs, GCNII, GCNII*, FB-GCNII, and FB-GCNII* use λ = 1 .and α = 0 .
2 on Actor and Squirrel .
Other deep models use the same training configurations as
GCNII [9] on the remaining 7 datasets.
For all experiments, we use the same 48%/ 32%/ 20% splits for training, validation and testing as in [44].
We report the average performance of all models on the test sets over 10 splits 8 .
We tune the learning rate in {0.01, 0.05, 0.1}, weight decay in {0, 5e-6, 1e-5, 5e-5, 1e-4, 5e-4, 1e-3, 5e-3, 1e-2}, and dropout in {0, 0.1, 0.2, ..., 0.9}.
[Figure]
[Table]
Theresultsareaveragedfrom 10 independentruns.
The(values)representthedifferenceofperformancebroughtbypatchingFB
[Table] [Table]
5.2 Supervised Learning of Shallow GNNs
In Table
[Table]
we summarize the mean accuracy of shallow baseline GNNs and their filterbank versions.
The best performance is highlighted.
Also, we record the performance differences between baselines and the two-channel augmented methods in the brackets.
For better intuitive understanding of the effectiveness of incorporating the high-pass filter, we present in Figure 2 the t-SNE visualization of the learned embedding in different channels of FB-GCN for Squirrel dataset.
Those of the other datasets will be provided in the appendix D.
From the results we can see that, our propose methods generally boost the performance of almost all cases, especially when the labels are not much smoother than the input features indicated, considering the S -values in Table 3.
Table 3 shows that the S -values of FB-GCN outputs are closer to the S -values of the ground truth labels (see the absolute differences in the bracket) compared with those of GCN outputs.
This indicates that FB-GCN is able to learn better representations which can reconstruct both the smooth and non-smooth part of the ground truth labels.
Note that we measure the smoothness by ˆ L sym instead of L sym , because GCN is train with renormalized affinity matrix ˆ A sym .
5.3 Supervised Learning of Deep GNNs
In this subsection, we build deep multi-hop filterbank models based on the architecture of GCNII and
GCNII* [9] to see if the two-channel method is capable to assist deep GNN models.
We report mean accuracy, highlight best performing depth, and record performance difference in brackets in Table 4.
In general, FB-GCNII and FB-GCNII* achieve better results than the unpatched GCNII and GCNII* at different depths, especially on Wisconsin and Texas , where the non-smooth part of representations are desirable.
[Table]
[Table]
5.4 Ablation Tests
In this subsection
[Table]
we perform ablation tests by using FB-GraphSAINT [57] on Cora , Cornell and
Texas accordingly, which are the three principally different datasets measured by ˆ L sym in Table 3 .
The ablation tests would examine the effectiveness of each proposed component.
The results are summarized in Table 5.
The results show that both the non-linear feature extractor and two-channel filtering architecture are able to help capture richer information under different S -value distributions of input features and output labels.(See more ablation results in Appendix C.3)
Moreover, to emphasize the importance of HP component, we also test the learnable coefficients α L and α H for two components on FB-GraphSAINT over the 9 datasets at the validation stage (see
Table 7 in Appendix C.1).
For most of the tasks, neither the coefficients for LP nor HP are negligible.
Among 6 out of 9 tasks, the learned coefficients for the HP components are even greater, this indicates the necessity of the HP components in graph representation learning.
In addition, from the averaged real-time change of the learned coefficients α L and α H in the output layer of FB-GraphSAINT on Cora , Cornell and Texas during training (see Figure 3 in Appendix
C.2), we can see that α L and α H will converge to a pair of values that explains how the smooth and non-smooth features will be mixed.
The fact that the ratio α H /α
L is close to 1 again shows that the importance of the non-smooth part in constructing the output signal.
More specifically, the importance (red line) is higher when the demand of non-smooth outputs (diff values) is higher.
See more ablation study of GCN on PPI in Appendix C.3.
Conclusion
This paper recognizes the role of high-frequency information in graph representation learning.
The proposed HP filter completes the spectrum of graph filters and yield significantly better representations on several empirical tasks.
The importance of the non-smooth component in graph signals can be revealed by the new defined S-value.
A Dataset Descriptions
For node classification.
there are 4 main categories:
Cora , Citeseer , and Pubmed are 3 benchmark datasets [47] in the category of Citation network .
Such networks use nodes to represent papers and edges to denote citations.
Node features are the bag-of-words representation and node labels are classified into different academic topics.
Cornell , Texas , and Wisconsin belong to the webpage dataset WebKB [44] created by Carnegie Mellon
University.
Each node represents a web page, and the edges are hyperlinks between nodes.
Node features are the bag-of-words representation and node labels are in five classes.
Chameleon and Squirrel are twi page-to-page networks in the Wikipedia network [46].
Nodes represent web pages and edges show mutual links between pages.
Node features are informative nouns in the Wikipedia pages and nodes are classified into 5 groups based on monthly views.
Actor refers to the Actor co-occurrence network .
A node correspond to an actor, and an edge exists if two actors occur on the same Wikipedia page.
Node features correspond to some keywords in the
Wikipedia pages, and nodes are categorized into five classes of words of actors actor’s Wikipedia.
B Graph Classification
For the graph classification tasks, we compare the patched methods FB-GIN- 0 and FB-GIN- ϵ against the baselines GIN- 0 and GIN- ϵ , with the same experiment setting as [56].
The results (accuracy and standard deviation) are provided in Table 6.
C More Ablation Tests C.1 Ablation Coefficients C.2 Real Time Change of Coefficients
[Figure]
C.3 Ablation Tests on PPI
D t-SNE Visualization
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure] [Figure]
E Discussion of Filters E.1 Lazy Random Walk Matrix
In graph signal processing, lazy random walk defined in the following equation is often used as a LP filter [14],
[Equation]
It can be seen as adding D ii self-loops to the i -th node of A and normalize it to be a random walk matrix and 0 ≤ λ i ( A lrw ) ≤ 1 .
Such spectral property makes it a standard band-pass filter, which can avoid some theoretical confusion due to negative eigenvalues.
Unlike ˆ A rw , A lrw maintains certain topology properties of A rw , e.g., stationary distribution and eigenvectors.
In practice, these properties are supposed to be changed, unless one has strong prior knowledge that GNNs will benefit from the renormalized one. Furthermore, it is found that adding self-loops can shrink the magnitude of the dominant eigenvalue so that the influence of long-distance nodes will be reduced, which makes the filtered signal more dependent on local information [21]. There is growing empirical evidence showing that adding self-loops will lead to effective graph convolutions on some applications [29, 53]. Compared to ˆ A rw ,
A lrw works better at reducing the magnitude of dominant eigenvalue under certain conditions. We show it in the following theorem. Theorem 1. We denote the generalized lazy random walk matrix and the generalized renormalized adjacency matrix respectively by
[Equation]
where ˜
A γ = γI + A, ˜ D γ = γI + D .
Suppose G has no isolated node, i.e., D ii > 0 for all i , and for positive γ , we have
[Equation]
where λ 1 ( · ) and λ 2 ( · ) are the largest and second largest eigenvalues a matrix.
Proof.
{+Detailed proof can be found in Appendix E.2. The HP filer derived from A lrw is ( I − A rw ) / 
 and they can be used as a set of filterbank in FB-GNN framework.+} From table 9 we can see that, FB-GNNs with lazy random walk matrix can boost the performance of baseline GNNs more significantly than symmetric renormalized affinity matrix on heterophilic datasets Cornell, Wisconsin, Texas and Film , where baseline GNNs underperform MLP.
And on homophilic datasets, where baseline GNNs outperform MLP, FB-GNNs with symmetric renormalized affinity matrix perform better. [Table] [Table]
E.2 Proof of Eigengap
Proof.
Denote the symmetric normalized lazy random walk matrix and the generalized symmetric renormalized adjacency matrix respectively by
[Equation]
It is easy to verify that
[Equation]
where λ ( · ) denotes the spectrum of a matrix.
Since λ 1 ( A γ lrw ) =
λ 1 ( ˆ A γ rw )
= 1 , to prove the theorem, it is necessary and sufficient to prove
[Equation]
It is easy to show that D 1 / 2 1 is an eigenvector of A γ slrw corresponding to λ 1 ( A γ slrw ) and ˜ D 1 / 2 γ 1 is an eigenvector ˆ A γ sym corresponding to λ 1 ( ˆ A γ sym ) .
By the Rayleigh quotient theorem ([25]),
[Equation]
By the Rayleigh quotient theorem and the Courant-Fischer min-max theorem ([25]),
[Equation]
Then from (12) and (13) we obtain
[Equation]
In this subsection, we report the optimal hyperparameters that are searched for FB-GNNs
[Equation] [Table]
[Table]
Computers
[Table]