D EMYSTIFYING G RAPH N EURAL N ETWORKS VIA G RAPH F ILTER A SSESSMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Neural Networks (GNNs) have recently received tremendous attention due to their power in handling graph data for different downstream tasks across different application domains.
Many GNN models have been proposed, which mainly differ in their graph ﬁlter design.
However, most of these models believe there is a best ﬁlter for all the graph data. Instead, we attempt to provide in-depth analysis on (1) Whether there exists an optimal ﬁlter that performs the best on all graph data; (2) Which graph properties should be considered for ﬁnding the optimal graph ﬁlter; and (3) How to design appropriate ﬁlters that adapt to a given graph.
In this paper, we focus on addressing the above three questions for the semi-supervised node classiﬁcation task.
We propose a novel assessment tool, called Graph Filter Discriminant Score (GFD), for evaluating the effectiveness of graph convolutional ﬁlters for a given graph in terms of node classiﬁcation.
Using the assessment tool, we ﬁnd out that there is no single ﬁlter as a “silver bullet” that performs the best on all possible graphs, and graphs with different properties are in favor of different graph convolutional ﬁlters.
Based on these ﬁndings, we develop Adaptive Filter Graph Neural Network (AFGNN), a simple but powerful model that can adaptively learn data-speciﬁc ﬁlters.
For a given graph, AFGNN leverages graph ﬁlter assessment as regularization an extra loss term and learns to combine a set of base ﬁlters.
Experiments on both synthetic and real-world benchmark datasets have demonstrated that our proposed model has the ﬂexibility in learning an appropriate ﬁlter and consistently provides state-of-the-art performance across all the datasets.
I NTRODUCTION
Graph Neural Networks (GNNs) are a family of powerful tools for representation learning on graph data, which has been drawing more and more attention over the past several years.
GNNs can obtain informative node representations for a graph of arbitrary size and attributes, and has shown great effectiveness in graph-related down-stream applications, such as semi-supervised node classiﬁcation (Kipf & Welling, 2017), graph classiﬁcation (Wu et al., 2019b), graph matching (Bai et al., 2019; Li et al., 2019), recommendation systems (Ying et al., 2018), and knowledge graphs (Schlichtkrull et al., 2018).
As GNNs have superior performance in graph-related tasks, the question as to what makes GNNs so powerful is naturally raised.
Note that GNNs adopt the concept of the convolution operation into graph domain.
To obtain a representation of a speciﬁc node in a GNN, the node aggregates representations of its neighbors with a convolutional ﬁlter.
For a task related to graph topology, the convolutional ﬁlter can help GNN nodes to get better task-speciﬁc representations (Xu et al., 2019).
Therefore, it is the ﬁlter that makes GNNs powerful, and thus the key to designing robust and accurate GNNs is to design proper graph convolutional ﬁlters that can best leverage the graph topology for a given graph.
Recently, many GNN architectures are proposed (Kipf & Welling, 2017; Velickovic et al., 2018; Xu et al., 2019; Wu et al., 2019a; Maehara, 2019), with their own graph ﬁlter designs.
However, none of them have properly answered the following fundamental questions of GNNs: (1) Is there a best ﬁlter that works for all graphs?
(2) If not, what are the properties of graph structure that will inﬂuence the performance of graph convolutional ﬁlters?
(3) Can we design an algorithm to adaptively ﬁnd the appropriate ﬁlter for a given graph?
In this paper, we focus on addressing the above three questions for semi-supervised node classiﬁcation task.
Inspired by studies in Linear Discriminant Analysis (LDA), we propose a Graph Filter
Discriminant (GFD) Score metric to measure the power of a graph convolutional ﬁlter in discriminating node representations of different classes on a speciﬁc graph.
We have analyzed all the existing
GNNs’ ﬁlters with this assessment method to answer the three aforementioned questions.
We found that no single ﬁlter design can achieve optimal results on all possible graphs.
In other words, for different graph data, we should adopt different graph convolutional ﬁlters to achieve optimal performance.
We then experimentally and theoretically analyze how graph structure properties inﬂuence the optimal choice of graph convolutional ﬁlters.
Based on all of our ﬁndings, we propose the Adaptive Filter Graph Neural Network (AF-GNN), which can adaptively learn a proper model for the given graph.
We use the Graph Filter Discriminant
Score (GFD) as a regularization an extra loss term to guide the network to learn a good data-speciﬁc ﬁlter, which is a linear combination of a set of base ﬁlters.
We show that the proposed Adaptive Filter can better capture graph topology and separate features on both real-world benchmark datasets and synthetic datasets.
We highlight our main contributions as follows:
• We propose an assessment tool: Graph Filter Discriminant Score, to analyze the effectiveness of graph convolutional ﬁlters.
Using this tool, we ﬁnd that no best ﬁlter can work for all graphs, and that the optimal choice of a graph convolutional ﬁlter depends on the graph data.
• We propose Adaptive Filter Graph Neural Network that can adaptively learn a proper ﬁlter for a speciﬁc graph using the GFD Score as guidance.
• We demonstrate that the proposed model can ﬁnd better ﬁlters and achieve better performance compared to existing GNNs, on both real-word and newly created benchmark datasets.
P RELIMINARIES
Semi-Supervised Node Classiﬁcation.
Let Y be the class assignment vector for all the nodes in V .
C indicates the total number of classes, and Y v ∈ { 1 , · · · , C } indicates the class that node v belongs to.
The goal of semi-supervised node classiﬁcation is to learn a mapping function f : V → { 1 , · · · , C } using the labeled nodes, and predict the class labels for the unlabeled nodes, i.e., ˆ Y v = f ( v ) , by leveraging both node features X and graph structure A .
Graph Data Generator.
Intuitively, semi-supervised node classiﬁcation requires both node features ( X ) and the graph structure ( A ) to be correlated to the intrinsic node labels ( Y ) to some extent.
To systematically analyze the performance of different GNN ﬁlters, we test their performance under different graph data with different properties, i.e., graphs with different X , A , Y .
Intuitively, both graph topology and node features have to be correlated with the node labels, if including both can enhance the performance of node classiﬁcation task.
To better understand the roles played by each component, we assume the graphical model to generate a graph data is as described in Fig. 1(a).
To better disclose the relationship between different graph ﬁlters and properties of different graph data, we further make assumptions on how X and A are generated when Y is given, as it is difﬁcult to obtain those properties from real-world data.
Therefore, we study simulated data to support a thorough analysis.
We now describe the generation of Y , X | Y , and A | Y respectively.
Generating Y :
Each node is randomly assigned with a class label with probability proportional to its class size.
We assume each class c is associated with n c nodes, which are indexed based on their class groups.
Generating X | Y :
We assume that node features are sampled from a distribution determined by their corresponding labels.
For example, we can sample node features of class c from a multivariate
Gaussian distribution with the parameters conditioned on class c : X ( c ) ∼ N ( µ ( c ) , Σ ( c ) ) .
For another example, we can sample node features of class c from a circular distribution with radius r c and noise noise c conditioned on c.
Generating A | Y : We follow the most classic class-aware graph generator, i.e. stochastic block model (SBM, Holland et al.
(1983)), to generate graph structure conditioned on class labels.
SBM has several simple assumptions that (1) edges are generated via Bernoulli distributions independently and (2) the parameter of the Bernoulli distribution is determined by the classes of the corresponding pair of nodes v i and v j , i.e., A ij | Y
i , Y j ∼ Ber ( p Y i Y j ) , where p Y i
Y j is a parameter determined by the two corresponding classes.
In a simple two-class case, p = p 11 = p 22 denotes the probability that the linked pair belongs to the same class, while q = p 12 = p 21 denotes the probability that the linked pair belongs to different classes.
We call p + q 2 the “density of graph” , which controls the overall connectivity of a graph, and we call | p − q | the “density gap” , which controls how closely the graph generated by SBM correlates with labels.
We assume p ≥ q in all the following sections.
Degree Corrected SBM (DCSBM, Karrer & Newman (2011)), which is a variation of
SBM, adds another parameter γ to control the “power-law coefﬁcient” of degree distribution among nodes.
Figure 1(b-e) demonstrates examples of synthetic graphs generated by SBM and DCSBM with different graph structure properties.
[Equation]
) Dependency of
Y , X and A [Equation] (c
[Figure] (d) DCSBM: p = 0 . 3 , q = 0 . 05 , γ = − 0 .
[Figure]
Graph Convolutional Filters.
By examining various GNN designs, we ﬁnd that most of the GNN operators can ﬁt into a uniﬁed framework, i.e., for the l -th layer:
[Equation]
which describes the three-step process that involves: (1) a graph convolutional operation (can also be regarded as feature propagation or feature smoothing) denoted as F ( G ) H ( l − 1) ,
(2) a linear transformation denoted by multiplying W , and (3) a non-linear transformation denoted by σ ( · ) .
Clearly, the graph convolutional operation F ( G ) H ( l − 1) is the key step that helps GNNs to improve performance.
Thus, to design a good GNN, a powerful graph convolutional ﬁlter F ( G ) is crucial.
We analyze the effectiveness of graph ﬁlters for existing GNNs in the following.
The work of GCN (Kipf & Welling, 2017)
ﬁrst adopts the convolutional operation on graphs and use the ﬁlter F ( G ) =
˜ D − 1 / 2 ˜
A ˜ D − 1 / 2 .
Here, ˜ A
=
A
+ I is the self-augmented adjacency matrix, and
˜ D
= diag ( ˜ d 1 , ..., ˜ d n ) is the corresponding degree matrix, where ˜ d i = (cid:80) nj
=1 ˜
A ij .
Some studies (Wu et al., 2019a; Maehara, 2019) use a ﬁlter F ( G ) =
( ˜ D − 1 / 2 ˜
A ˜ D − 1 / 2 ) k
that is similar in form to GCN’s ﬁlter, but with a pre-deﬁned exponent k greater than one.
This would help a node to obtain information from its further neighbors without redundant computation cost.
Several other studies propose to use sampling to speed up GNN training (Chen et al., 2018b; Hamilton et al., 2017; Chen et al., 2018a)), which can be considered as a sparser version of GCN’s ﬁlter.
Another set of GNNs consider using a learnable graph convolutional ﬁlter.
For example, Xu et al.
(2019) and Chiang et al.
both propose to use F ( G ) =
A + (cid:15) I where (cid:15) is a learnable parameter to augment self-loop skip connection.
Graph Attention Networks(Velickovic et al., 2018) proposes to assign attention weight to different nodes in a neighborhood, which can be considered as a ﬂexible learnable graph convolutional ﬁlter.
Their graph ﬁlters applied on a feature matrix X can be considered as: ∀ i, j , F ( G ) ij = [Equation] |
[Equation]
|| [Equation] [Equation] A ij , where N i is the neighborhood of node i , α is a learnable weight vector, and || indicates concatenation.
A SSESSMENT T OOL FOR A NALYZING G RAPH C ONVOLUTIONAL F ILTERS
In this section, we introduce a novel assessment tool for analyzing graph convolutional ﬁlters.
We ﬁrst review the Fisher score, which is widely used in Linear Discriminant Analysis to quantify the linear separability of two sets of features.
With the Fisher score, we propose the Graph Filter
Discriminant Score metric to evaluate the graph convolutional ﬁlter on how well it can separate nodes in different classes.
3. T HE A SSESSMENT T OOL : G RAPH F ILTER D ISCRIMINANT S CORE
Fisher Score.
When coming to traditional non-graph data, the Fisher Score (Fisher, 1936) is widely used to assess the linear separability between two classes.
Given two classes of features X ( i ) and
X ( j ) , the Fisher Score is deﬁned as the ratio of the variance between the classes (inter-class distance) to the variance within the classes (inner-class distance) under the best linear projection w of the original feature:
[Equation]
where µ ( i ) and µ ( j ) denotes the mean vector of X ( i ) and X ( j ) respectively, Σ ( i ) and Σ ( j ) denotes the variance of X ( i ) and X ( j ) respectively, and w denotes the linear projection vector which we can understand as a rotation of the coordinate system, and the max w operation is to ﬁnd the best direction in which these two class of nodes are most separable.
As the numerator of J indicates interclass distance and the denominator of J indicates inner-class distance a larger value of J indicates higher separability.
Note that for given features, we can directly get the closed form solution of the optimal w , with which Fisher Score could be deformed as: J ( X ( i ) , X ( j ) )
= ( µ ( i ) − µ ( j ) ) (cid:62) ( Σ ( i ) + Σ ( j ) ) − 1 ( µ ( i ) − µ ( j ) ) .
The detailed proof is provided in appendix A.2.
We expand Fisher Score to multiple classes by considering a weighted sum of Fisher Score of each pair of classes.
Graph Filter Discriminant Score.
As mentioned before, the key component that empowers GNNs is the graph convolutional ﬁlter F ( G ) .
Intuitively, an effective ﬁlter should make the representations of nodes in different classes more separable.
Therefore, we propose to use Fisher Scores of the node representations before and after applying the graph convolutional ﬁlter in order to evaluate this ﬁlter.
For each pair of classes ( i, j ) , we compare the difference of their Fisher
Score of initial representations J ( X ( i ) , X ( j ) ) and their Fisher Score of representation after ﬁlter J (cid:16)(cid:0) F ( G ) X (cid:1) ( i ) , (cid:0) F ( G ) X (cid:1) ( j ) (cid:17) . We deﬁne such difference as “Graph Filter Discriminant Score” for the ﬁlter F ( G ) with respect to data X as follows. (cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40) (cid:40) (cid:40)(cid:40) (cid:40)(cid:40) (cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40)(cid:40) (cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)
[Equation] (cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104)(cid:104) GFD
[Equation] 3) where n i and n j is the node size within i and j class, ( k, t ) denotes all possible class pairs, and n i + n j (cid:80) k (cid:54) = t ( n k + n t ) serves as normalization for classes with different node numbers.
For each pair of classes ( i, j ) , we deﬁne their Fisher Difference as FD ( i, j ) = J (cid:0)(cid:0) F ( G ) X (cid:1) ( i ) , (cid:0) F ( G ) X (cid:1) ( j ) (cid:1) −
J (cid:0) X ( i ) , X ( j ) (cid:1) , which is the difference of their Fisher Score of representations after applying the ﬁlter F ( G ) and their Fisher Score of initial representations.
We then deﬁne the GFD Score for the ﬁlter F ( G ) with respect to feature matrix X as follows:
[Equation]
where n c is the number of nodes in class c .
Note that the GFD Score is a weighted sum of the Fisher
Difference for each pair of classes.
Intuitively, the larger the GFD score, the more effective is this corresponding ﬁlter to increase the separability of node features.
The Fisher Score can be extended to evaluate non-linearly separable data in addition to linearly separable data.
We claim the rationale of such measure by showing that the graph convolution can actually help non-linearly separable data to be linearly separable if the graph ﬁlter is chosen properly for a given graph.
As shown in Figure 2(a) ∼ (d), if we use a proper ﬁlter, the convolutional operation can transform three circular distributions, which are non-linearly separable, into three linearly separable clusters.
Moreover, as shown in Figure 2(e) ∼ (h), even if the original features of different classes are sampled from the same distribution, the proper graph convolutional ﬁlter can help to linearly separate the data.
This phenomenon shows that if the graph structure ( G ) is correlated with the task ( Y ), a proper ﬁlter alone is powerful enough to empower GNNs with nonlinearity, without any non-linear activation.
This phenomenon is also supported by the promising result of SGC (Wu et al., 2019a), which removes all the non-linear activations in the original GCN architecture.
Therefore, we claim that the proposed GFD is a reasonable metric to evaluate a graph ﬁlter’s effectiveness, and a good graph ﬁlter for a given graph should have a higher GFD score on that graph. [Figure]
3. A SSESSING E XISTING G RAPH C ONVOLUTIONAL F ILTERS
With the help of the assessment tool [Figure] we now examine existing ﬁlters and try to answer the two fundamental questions: (1) Is there a best ﬁlter that works for all graphs?
(2) If not, what are the properties of graph data that will inﬂuence the performance of graph convolutional ﬁlters?
The GFD Score we introduced in the above section can be applied to any ﬁlter on any given graph.
From Table 3, we can see that most of the current GNNs fall into the following ﬁlter family: { ( ˆ A ) k } , where the base ˆ A is a normalized adjacency matrix, and k is the order of the ﬁlter.
Note that there are some other variants of GNN ﬁlters that do not fall into this family, for example, GAT, but the analysis is similar.
Without loss of generality, we focus on analyzing this ﬁlter family.
The two main components of this ﬁlter family are the normalization strategy ( ˆ A ) and the order to use ( k ).
For simplicity, we study the roles of these two components separately, using our assessing tool to show whether there exists an optimal choice of ﬁlter for different graph data.
If an optimal choice does not exist, we determine the factors that will inﬂuence our choice of component.
Through the analysis, we choose SBM and DCSBM introduced previously to generate the structures of synthetic graphs, and choose multivariate Gaussian distributions to generate features of synthetic graphs.
We focusing on the structure properties that inﬂuence the optimal choice of ﬁlter.
We enumerate the hyper-parameters to generate graphs with different structure properties, including the power law coefﬁcient ( γ ) that controls the power law degree distribution of the graph, label ratio ( n 1 n 2 ) that indicates how balance are the classes of this graph, density ( p + q 2 ) that indicates the overall connectivity of the graph, and density gap ( | p − q | ) that indicates structural separability of the graph.
are
Analyzing Filter’s Normalization Strategy.
We consider three normalization strategies, including row normalization D − 1 A , column normalization AD − 1 , and symmetric normalization
D − 1 / 2 AD − 1 / 2 .
We calculate GFD scores of these three graph ﬁlters for graphs generated with different parameters.
As shown in Figure 3, no single normalization strategy is optimal for all graphs.
Here we give an empirical explanation to this phenomenon.
Note that, with the same order, each ﬁlter has the same receptive ﬁeld, and different normalization strategies affect only on how to assign weights to the neighboring nodes.
The row normalization strategy simply takes the mean of features of the node’s neighbors.
Clearly, this would help to keep every node’s new representations in the same range.
On the contrary, column normalization and symmetric normalization, might keep a larger representation for higher-degree nodes.
Using a column-normalized adjacency matrix as the base of the graph convolutional ﬁlter is similar to the PageRank algorithm.
While a node propagates its features to neighbors, this normalization strategy takes its degree into account.
Thus, column normalization can be helpful when the when node degree plays an important role for classiﬁcation.
Symmetric normalization combines the properties from both the row normalization and the column normalization.
Even in the case where row normalization and column normalization do not perform well, symmetric normalization still leads to promising performance.
Since there is no a single “silver bullet”, we now examine which graph properties inﬂuence our choice of the optimal normalization strategy, which may vary per graph.
We ﬁnd that power law coefﬁcient γ is a very important factor that inﬂuences the choice of normalization strategy.
As shown in Figure 3, when power-law coefﬁcient γ decreases (which indicates the graph’s extent of power-law) , row normalization tends to have better performance.
This is because row normalization helps to keep node representations in the same range, so that large representations of high degree nodes can be avoided.
Therefore, it prevents nodes with similar degrees getting closer to each other and messing the classiﬁcation tasks where node degrees are not important.
We also ﬁnd that the label ratio ( n 1 n 2 ) matters.
As shown in Figure 3, when the size of each class becomes more imbalanced, column normalization tends to work better.
This is because column normalization better leverages degree property and can help nodes in large-size class obtain larger representations, and thus the representations for nodes within different classes would become more separableduring representation smoothing, as nodes in large-size classes tend to have larger representation since they are more likely to have higher degree.
This can help nodes within different classes become more separable.
Analyzing Filter’s Order.
We then analyze what would be the best order for ﬁlters.
With a highorder ﬁlter, a node can obtain information from its further neighbors, and thus the amount of information it receives during the feature propagation increases.
But do we always need more information under any circumstances?
The answer is no.
Still, we ﬁnd that, for different graphs, the order that results in the best performance would be different 1 .
Since there is no best ﬁlter order for all the cases, we explore the factors that can inﬂuence the choice of order.
We ﬁnd that the density of graph and density gap between two classes have a big impact.
As shown in Figure 4, when the density or density gap increases, the ﬁlter with higher order tends to be a better choice.
We provide an intuitive explanation for this phenomenon as follows.
Note that the feature propagation scheme is based on the assumption that nodes in the same class have a closer connection.
On one hand, when the density increases, the connections between nodes are closer.
Therefore, high-order ﬁlters can help gather richer information and thus reduce the variance of the obtained new node representations, so it helps nodes in the same class get smoother representations.
On the other hand, when the density gap decreases, for a node, the size of neighbors within the same class becomes similar to the size of neighbors within different classes.
Thus conducting high-order graph convolution operations will mix the representations of all nodes regardless of classes, which will make node classiﬁcation more difﬁcult.
Thus, for graphs with a small density gap, low-order ﬁlters are preferred; and for a graph with a large density gap, high-order ﬁlters are preferred.
L EARNING TO F IND THE O PTIMAL G RAPH C ONVOLUTIONAL F ILTER
Based on previous analysis, we now answer the last question: Can we design an algorithm to adaptively ﬁnd the appropriate ﬁlter for a given graph?
We develop a simple but powerful model, the
Adaptive Filter Graph Neural Network (AFGNN).
For a given graph, AFGNNs can learn to combine an effective ﬁlter from a set of ﬁlter bases, guided by GDF Scores.
Adaptive Filter Graph Neural Network (AFGNN).
For simplicity, we only consider ﬁnding the optimal ﬁlter for one family of graph convolutional ﬁlters:
[Equation]
where k is the maximum order.
Note that, we also include the identity matrix, which serves as a skip-connection, to maintain the original feature representation.
Based on our previous analysis, for graphs that are not closely correlated to tasks (i.e., small density gap in SBM), the identity matrix will outperform all the other convolutional ﬁlters.
We denote the above 3 k + 1 ﬁlters as
F base( G ) , · · · , F base 3 k +1 ( G ) , the l -th layer of AFGNN is deﬁned as a learnable linear combination of these ﬁlter bases:
[Equation]
where α ( l ) is softmax-normalized vector gotten from ψ ( l ) to combine base ﬁlters. where ψ ( l ) is the learnable vector to combine base ﬁlters and α ( l ) is its softmax-normalized version.
Comparing to GNNs with ﬁxed ﬁlters such as GCN and SGC, our proposed AFGNN can adaptively learn a ﬁlter based on any given graph.
As we have shown that no single ﬁxed ﬁlter can perform optimally for all graphs, we conclude that an adaptive ﬁlter has more capacity to learn better representations.
Comparing to other GNNs with learnable ﬁlters such as GAT, AFGNN is computationally cheaper and achieves similar or better performance on most existing benchmarks and our synthetic datasets (as shown in the experiment section).
we can also consider adding complex ﬁlter as GAT into our ﬁlter bases.
We leave expanding the base ﬁlter family and adding more complex ﬁlters such as GAT into our ﬁlter bases as future work.
Training Loss.
To train this AFGNN model, we can simply optimize the whole model via any downstream tasks, i.e., node classiﬁcation.
However, as most of the semi-supervised node classiﬁcation datasets only contain limited training data, the enlarged ﬁlter space will make the model prone to over-ﬁtting.
Thus, we decide to add the GFD Score as a regularization an extra loss term into the training loss:
[Equation]
, where L
[Equation]
where L CE is the cross-entropy loss of the node classiﬁcation, and L GFD is deﬁned as the cumulative negation of GFD Score for the learned adaptive ﬁlter F AFGNN ( G ) ( l ) at each layer with respect to its input representation H ( l − 1) .
Noted that the GFD score assume a ﬁxed data representation, and is invariant to linear transformation, thus it’s not reasonable to use it for optimizing parameters in previous layers or the linear weights.
So, we only use GFD to optimize the ﬁlter choice, i.e., ψ .Note that the GFD score is invariant to linear transformation, so we mainly want to use GFD as a guidance to optimize the ﬁlter weights, i.e., ψ ( l ) , and do not count on it much to optimize linear transformation weights.
During the training, we minimize L to learn the proper model.
With a different choice of the regularization term λ weight λ for GFD loss, we can categorize our model into:
AFGNN 0 :
With λ = 0 , the model is only trained by classiﬁcation loss, which might be prone to over-ﬁtting when data is not sufﬁcient.
AFGNN 1 : With λ = 1 , the model is trained by both classiﬁcation loss and regularization loss, and the learned ﬁlter is a linear combination of base ﬁlters.
AFGNN ∞ :
With λ = ∞ , we implement it iteratively: after the model is optimized with L CE , we directly ﬁnd the optimal combination α ( l ) of base ﬁlter with highest GFD score for each layer l , which is equivalent to applying an inﬁnite regularization.
Note that H (0) is the input feature, we can simply pre-compute the best ﬁlter for the ﬁrst layer and keep it ﬁxed.
AFGNN 1 : With λ = 1 , the model is trained by both classiﬁcation loss and GFD loss simultaneously.
This can help learn a better ﬁlter which is a linear combination of base ﬁlters.
AFGNN ∞ :
This case is not exactly λ = ∞ , and the training process is different from the other two cases.
We implement the training iteratively: we optimize the combination of base ﬁlters by training only with GFD loss L GFD , then we optimize the linear transofrmation parameter W l s with classiﬁcation loss L CE .
Note that the input feature H (0)
=
X is invariant, we can pre-train the optimal ﬁlter for ﬁrst layer and ﬁx it.
E XPERIMENTS
In this section, we evaluate whether AFGNN can learn an effective ﬁlter and perform well for node classiﬁcation.
Dataset We ﬁrst evaluate AFGNN on three widely used benchmark datasets: Cora, Citeseer, and
Pubmed (Sen et al., 2008).
As these datasets are not sensitive enough to differentiate the models, we need more powerful datasets that can evaluate the pros and cons of each model.
Based on our ﬁndings in section 3.2, we generate two synthetic benchmarks called SmallGap and SmallRatio.
SmallGap corresponds to the case in which the density gap of the graph is close to 1.
This indicates that the graph structure does not correlate much to the task, thus I would be the best ﬁlter in this case.
SmallRatio corresponds to the case in which the label ratio is small, i.e. the size of one class is clearly smaller than the other, and column normalization AD − 1 is the best normalization 2 .
Baselines and Settings.
We compare against 5 baselines, including GCN, GIN, SGC, GFNN, and
GAT.
To make fair comparisons, for all the baseline GNNs, we set the number of layers (or orders) to be 2, and tune the parameters including learning rate, weight decay, and number of epochs 3 .
For all the benchmark datasets, we follow the data split convention 4 .
For the synthetic dataset, we conduct 5-fold cross-validation, randomly split the nodes into 5 groups of the same size, take one group as the training set, one as the validation set and the remaining three as the test set.
Each time we pick the model with highest validation accuracy and record its test accuracy.
For each dataset, we run the experiment 10 times and compute the mean and standard deviation of recorded test accuracy. [Table] [Table]
Classiﬁcation Performance.
As is shown in Table 1, our proposed AFGNN ∞ model can consistently achieve competitive test accuracy.
On Pubmed, SmallGap, and SmallRatio, AFGNN ∞ can achieve the best results among all the baseline models.
On Cora and Citeseer, though GAT outperforms our proposed model a little bit, however, as shown in Table 6,7, GAT takes a longer time to train and converge, and has more memory cost as well.
Also, when the given graph is simple, GAT would suffer unavoidable overﬁtting problem.
We further compare our AFGNN 0 , AFGNN 1 , AFGNN ∞ to examine the role of GFD regularization loss.
The AFGNN 0 performs quite poorly on all the datasets, implying that the larger search space of the ﬁlter without regularization GFD loss is prone to over-ﬁtting, while AFGNN 1 and AFGNN ∞ perform much better.
Also, AGFNN ∞ has superior performance compared to AFGNN 1 , which indicates the GFD Score is indeed a very powerful tool for assessing the effectiveness of the ﬁlter.
Graph Filter Discriminant Analysis.
We are also interested to see whether the proposed method can indeed learn the best combination of ﬁlters from the base ﬁlter family.
To do so, we calculate the
GFD Score of the ﬁrst-layer ﬁlter learned by AFGNN 0 , AFGNN 1 , AFGNN ∞ and the seven base ﬁlters on the test set for each dataset.
For the AFGNN models, the ﬁlter is trained with the training set for each dataset.
Table 2 5 shows the GFD score and Figure 5 shows the ﬁlter combination learned by AFGNN ∞ , we can see that our proposed method can indeed learn a combined ﬁlter on all the datasets.
Speciﬁcally, in all the benchmark datasets, the best base ﬁlter is ( ˜ D − 1 ˜ A ) 2 , and our proposed adaptive ﬁlter not only picks out the best base ﬁlter but also learns a better combination.
For the two synthetic datasets, where I and ( ˜ A ˜ D − 1 ) 2 are the best ﬁlters, our algorithm can also learn to pick them out.
We thereby conclude that the proposed GFD regularization loss can help ﬁnd an appropriate ﬁlter for a given dataset.
C ONCLUSION
Understanding the graph convolutional ﬁlters in GNNs is very important, as it can help to determine whether a GNN will work on a given graph, and can provide important guidance for GNN design.
In our paper, we focus on the semi-supervised node classiﬁcation task.
We ﬁrst propose the Graph
Filter Discriminant Score as an assessment tool for graph convolutional ﬁlter evaluation, and then apply this GFD Score to analyze a family of existing ﬁlters as a case study.
Using this tool, we learn that no single ﬁxed ﬁlter can produce optimal results on all graphs.
We then develop a simple but powerful GNN model: Adapative Filter Graph Neural Network, which can learn to combine a family of ﬁlters and obtain a task-speciﬁc powerful ﬁlter.
We also propose to add the negative GFD
Score as a regularization an extra component to the objective function, it can act as a guidance for the model to learn a more effective ﬁlter.
Experiments show that our approach outperforms many existing GNNs on both benchmark and synthetic graphs
A A PPENDIX A. S UMMARY OF G RAPH F ILTERS FOR E XISTING GNN S .
Table 3 summarized the graph ﬁlters for existing GNNs.
A. P ROOF OF P ROPOSITION
Proof According to the conclusions in linear discriminant analysis, the maximum separation occurs when w ∝ ( Σ ( i ) + Σ ( j ) ) − 1 ( µ ( i ) − µ ( j ) ) .
Note that, when we want to apply this ﬁsher linear discriminant score in our problem, the linear transformation part in our classiﬁer (and also the linear transformation part in GNN) will help to ﬁnd the best w .
Thus, we can directly plug the optimum solution w ∗ =
c ( Σ ( i ) + Σ ( j ) ) − 1 ( µ ( i ) − µ ( j ) ) into this formula, here c is a scalar.
Then, we’ll have:
[Equation]
Thus we completed the proof.
A. N O B EST F ILTER FOR A LL G RAPHS A.3. E XAMPLES OF “N O B EST N ORMALIZATION S TRATEGY FOR A LL ”
Figure 6 provides two examples to show there is no best normalization strategy for all graphs.
For both examples, we ﬁx the order of ﬁlter to be 2.
The ﬁrst row shows a case in which row normalization is better than the other two.
The corresponding graph contains 2 classes of nodes with size 500.
The graph structure is generated by
DCSBM with p = 0 .
3 , q = 0 .
05 , power law coefﬁcient γ = − 0 .
The features for two classes satisfy multivariate distribution with an identity co-variance matrix, and with mean (0.2,0.2) and (0,0) respectively.
In this example, we can clearly see that with other two normalization strategy, some high-degree hubs show up in the upper right corner from both class, which is harmful for classiﬁcation.
We generate this example to illustrate the beneﬁt of row normalization because row normalization would be very helpful for a graph with power law degree distribution, which contains some nodes with unusually large degree (those nodes are called hubs), since it can help avoid those hubs obtaining larger representations and thus be mis-classiﬁed.
The second row shows a case in which column normalization is better than the other two.
The corresponding graph contains 2 classes of nodes with size 900 and 100 respectively.
The graph structure is generated by SBM with p = 0 .
3 , q = 0 .
The features for two classes satisfy multivariate distribution with an identity co-variance matrix, and with mean (-0.2,-0.2) and (0.2,0.2) respectively.
We generate this example to illustrate the beneﬁt of column normalization because under this case, we should consider taking more degree information into consideration.
Therefore, column normalization would be more helpful.
[Figure]
A.3. E XAMPLES OF “N O B EST O RDER FOR A LL ”
Figure 7 provides two examples to show there is no best order for all graphs.
For both examples, we ﬁx the normalization strategy to be row normalization, and varies order to be 2, 4, 6. The ﬁrst row shows a case in which small order is better than the large ones. The corresponding graph contains 2 classes of nodes with same size 500. The graph structure is generated by SBM with p = 0 .
215 , q = 0 .
The features for two classes satisfy multivariate distribution with an identity co-variance matrix, and with mean (0.5,0.5) and (0,0) respectively.
The second row shows a case in which large order is better than the smaller ones. The corresponding graph contains 2 classes of nodes with same size 500. The graph structure is generated by SBM with [Figure]
A.3. I LLUSTRATION OF G RAPH G ENERATOR FOR C URVES IN S ECTION
For the curves indicating how powerlaw coefﬁcient inﬂuence the choice of normalization in Figure 3, we generate the corresponding graphs structure by DCSBM with ﬁxed p = 0 .
3 , q = 0 .
2 and varies the powerlaw coefﬁcient from -0.3 to 0.
The graph contains two classes of nodes, and is of size 400 and 600 for each class respectively.
The feature for each class satisﬁes multivariate normal distribution with identity co-variance matrix, and with mean (0,0) and (0.2,0.2).
For the curves indicating how label ratio inﬂuence the choice of normalization in Figure 3, we generate the corresponding graphs structure by SBM with ﬁxed p = 0 .
3 , q = 0 . 1 and varies the label ratio.
The graph contains a total number of 1000 nodes in two classes.
The feature for each class satisﬁes multivariate normal distribution with identity co-variance matrix, and with mean (0,0) and (0.5,0.5).
For the curves indicating how density inﬂuence the choice of normalization in Figure 4, we generate the corresponding graphs structure by SBM with ﬁxed density gap p/q = 1 . 5 and varies the density by varying q.
The graph contains two classes of node of size 500.
The feature for each class satisﬁes multivariate normal distribution with identity co-variance matrix, and with mean (0,0) and (0.5,0.5).
For the curves indicating how density gap inﬂuence the choice of normalization in Figure 4, we generate the corresponding graphs structure by SBM with ﬁxed density p + q = 0 .
6 and varies the density gap.
The graph contains two classes of node of size 500.
The feature for each class satisﬁes multivariate normal distribution with identity co-variance matrix, and with mean (-0.2,-0.2) and (0.2,0.2).
A. F LOWCHART OF AFGNN FOR NODE CLASSIFICATION
The following ﬂowchart (Figure 8) describes the process of how a one-layer AFGNN tackle node classiﬁcation task.
A. F EATURE P ROPAGATION USING DIFFERENT FILTERS
We reduced the dimension of feature by t-SNE (Maaten & Hinton (2008)).
We annotate the ﬁlter and the GFD Score in title of each subﬁgure.
Note that, identity also corresponds to the initial feature.
The ﬁgure is the feature representation obtained after conduct graph convolution operation once with the corresponding ﬁlter.
[Figure]
[Figure]
A. D ATASET A.6. B ENCHMARK D ATASET
We use three benchmark dataset: Cora, Citeseer and Pubmed for the node classiﬁcation task.
Their statictics are in table4.
Beside number of nodes, edges, classes, the dimension of feature
[Figure]
and the data split strategy, we also show the class ratio variance , which can indicates if this dataset is imbalance or not, density gap , which indicates the dependency of structure and labels
[Figure]
and density , which indicates the overall connectivity of a graph.
We provide the degree distribution in Figure 10, and we can clearly ﬁnd that these benchmark datasets has power law degree distribution
[Figure]
[Table] [Table]
A.6. S YNTHETIC D ATASET
We also generated two synthetic datasets: SmallGap and SmallRatio.
For SmallGap, we use SBM to generate a two class network with p = 0 .
2 and q = 0 .
The density gap p/q is very small in this case.
They have the same number of nodes and both have 64 dimension features sampled from gaussian distributions with different mean and same variance.
For SmallRatio, we use SBM to generate a two class network, which has 200 nodes for one class and 800 nodes for the other.
This dataset is called SmallRatio because n 1 /n 2 = 0 .
25 is small.
Their 64 features are sampled from gaussian distributions with different mean and different variance.
The detailed generation process and parameter can be found in our code.
A. M ODEL H YPER P ARAMETERS
For GCN, SGC, GFNN, GAT, we directly use their public implementations.
For GIN, the initial code is not for node classiﬁcation task, so we modify their code following Xu et al.
(2019) to conduct experiments.
We tune the number of epochs based on convergence performance.
For learning rate and weight decay, we follows the parameter setting provides by the corresponding public implementations unless we ﬁnd better parameters.
The tuned parameters can be found in our code resource.
A. B ASELINE A CCURACY ON B ENCHMARK D ATASET
We report the accuracy of node classiﬁcation task for baseline models on Cora, Citeseer, and Pubmed provided by corresponding literature.
Since GIN (Xu et al., 2019) is not originally evaluated on node classiﬁcation task, we do not have the reported number here.
The results is in Table 5.
A. T IME AND M EMORY C OST C OMPARISON
Both our AFGNN model and GAT model have a learnable ﬁlter.
We provide time and memory complexity comparison on benchmark datasets here to compare these two models.
As shown in Table 6, GAT’s time cost is at least three times of AFGNN’s time cost on both Cora and Citeseer dataset.
As shown in Table 7, AFGNN’s memory cost on both Cora and Citeseer are
[Table] [Table]
[Table]
half of GAT’s memory cost. GAT does not have recorded time and memory cost for Pubmed dataset because it requires too much memory cost and is not able to run on GPU.
Therefore, AFGNN needs less time and memory cost than GAT.
A. S UPPLEMENTARY E XPERIMENT
We generate a real-world dataset with imbalanced classes to justify hard cases may exist in realworld datasets.
We download a large scale academic graph called Open Academic Graph (OAG), and choose two ﬁelds that have a large disparity in the number of papers: (1) History of ideas, which consists of 1041 papers; (2) Public history, which consists of 150 papers.
Obviously this two classes are imbalanced [Table] and fall in the large label ratio gap problem.
We run supplementary experiment on the generated OAG graph, the experiment setting remains the same as experiment settings for synthetic graphs.
Table 8 shows the experiment results.
To evaluate the models, we compare their F1 score for each class, the weighted average F1 score (micro F1), and the average F1 score (macro F1).
Our
AFGNN ∞ model shows superior performance on this dataset.