A P IECE - WISE P OLYNOMIAL F ILTERING A PPROACH FOR G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph Neural Networks (GNNs) exploit signals from node features and the input graph topology to improve node classiﬁcation task performance.
However, these models tend to perform poorly on heterophilic graphs, where connected nodes have different labels.
Recently proposed GNNs work across graphs having varying levels of homophily.
Among these, models relying on polynomial graph ﬁlters have shown promise.
We observe that solutions to these polynomial graph ﬁlter models are also solutions to an overdetermined system of equations.
It suggests that in some instances, the model needs to learn a reasonably high order polynomial.
On investigation, we ﬁnd the proposed models ineffective at learning such polynomials due to their designs.
To mitigate this issue, we perform an eigendecomposition of the graph and propose to learn multiple adaptive polynomial ﬁlters acting on different subsets of the spectrum.
We theoretically and empirically show that our proposed model learns a better ﬁlter, thereby improving classiﬁcation accuracy.
We study various aspects of our proposed model including, dependency on the number of eigencomponents utilized, latent polynomial ﬁlters learned and performance of the individual polynomials on the node classiﬁcation task.
We further show that our model is scalable by evaluating over large graphs.
Our model achieves performance gains of up to 10% over the state-of-the-art models and outperforms existing polynomial ﬁlter-based approaches in general.
Our anonymized code is available at https: tinyurl.com/PPGNN .
I NTRODUCTION
In this work, we are interested in the problem of classifying nodes in a graph.
In this problem, a graph and features for all the nodes in the graph are made available.
We are also given labels for a subset of the nodes in the graph.
The learning model utilizes this information to predict labels for the remaining nodes.
Graph Neural Networks (GNNs) perform well on such problems (Kipf & Welling, 2017).
GNNs predict a node’s label by aggregating information from its neighbours.
Thus, the performance of GNN is dependent on how well the given graph correlates with the node labels.
Characterizing the correlation between the graph and node labels is an active area of research.
Several metrics have been proposed including edge homophily (Abu-El-Haija et al., 2019a; Zhu et al., 2020), node homophily (Pei et al., 2020), class homophily (Lim et al., 2021).
All these metrics show that GNNs perform well when the graphs and node labels are positively correlated.
For example, in the simplest case, GNNs work well when the node and its neighbours share similar labels.
However, the performance can be poor if this criterion is not satisﬁed.
Several proposed GNN models attempt to be robust to the underlying correlation between graphs and labels.
Some approaches modify the aggregation mechanism (Pei et al., 2020; Zhu et al., 2020;
Kim & Oh, 2021), while other approaches propose to estimate and leverage the label-label compatibility matrix as a prior (Zhu et al., 2021).
More recent approaches have tackled this problem from a ﬁlter learning perspective (Bo et al., 2021; Chien et al., 2021).
These methods learn a ﬁlter function that operates on the eigenvalues of the graph.
With eigenvalues having frequency interpretations (Shuman et al., 2013), the ﬁlter function selectively accentuates and suppresses various frequencies as required by the task.
This process enables the model to learn better embeddings, which translates to improved performance.
Our interest lies in the area of modelling GNNs that can learn an effective ﬁlter.
Chien et al.
(2021) proposed to learn a polynomial ﬁlter.
This model achieves good performance gains on several datasets with varying correlations between graphs and labels.
However, we observe that learning a polynomial of a degree lower than the number of nodes can be seen as solving an over-determined system of equations.
Our observation suggests needing a higher-order polynomial to model richer or more complex frequency responses for certain datasets.
The design of the model proposed in Chien et al.
(2021) makes it ineffective in learning a higher-order polynomial.
It happens because the model attempts to overcome over-smoothing by giving smaller coefﬁcient values to larger powers.
In this paper, we address this issue and make the following contributions:
• Inspired by Chien et al.
(2021), we propose to learn a polynomial ﬁlter; however, we do so by learning a sum of polynomials over different subsets of eigenvalues.
Such modelling enables learning higher-order polynomials with several low order polynomials acting over the different subsets.
• Such modelling, however, requires an eigendecomposition of the graph, which can be expensive.
We leverage the model in Chien et al.
(2021) and the fact that there exist efﬁcient algorithms for computing top and bottom eigen components to propose a practically efﬁcient model.
• We present theoretical analysis that suggests that our sum of polynomials approach can approximate a latent optimal ﬁlter better than a single polynomial.
We also show that the space of learnable ﬁlters and graphs using our approach is larger than what is possible with Chien et al.
• Our experimental results show that our model performs better than state-of-the-art methods on several datasets, giving up to 10% gains.
These gains indicate that modelling the ﬁlter as a sum of polynomials is needed to approximate the latent ﬁlter better.
We also answer other research questions through ablative studies, including the importance of the top and bottom eigenvalues, the number of top and bottom eigenvalues needed, the number of polynomials needed, and the trade-offs involved.
The rest of the paper is organized as follows: We present related work in Section 2, problem setup and motivation for the approach in Section 3, details of the proposed approach with complexity and theoretical analysis in Section 4 and ﬁnally experimental results and empirical analysis in Section 5.
R ELATED W ORKS
In the recent times, Graph Neural Networks (GNNs) have become an increasingly popular method for semi-supervised classiﬁcation with graphs.
Bruna et al.
(2014) set the stage for early GNN models, which was then followed by various modiﬁcations (Defferrard et al., 2016; Kipf & Welling, 2017; Hamilton et al., 2017;
Veli ˇ ckovi ´ c et al., 2018).
Incorporating random walk information (AbuEl-Haija et al., 2019b;a; Li et al., 2018) gave further improvements in these models, but they still suffer from over smoothing.
Several proposed models attempt to overcome this problem (Klicpera et al., 2019; Lukovnikov & Fischer, 2021;
Chamberlain et al., 2021;
Yang et al., 2021).
Another line of research explored the question of which graphs worked well with GNNs.
The critical understanding was that the performance of GNN was dependent on the correlation of the graphs with the node labels.
Several approaches (Abu-El-Haija et al., 2019a; Zhu et al., 2020; Wang & Derr, 2021) considered edge homophily and proposed a robust GNN model by aggregating information from several higher-order hops.
Kim & Oh (2021) also considered edge homophily and mitigated the issue by learning robust attention models.
Pei et al.
(2020) talks about node homophily and proposes to aggregate information from neighbours in the graph and neighbours inferred from the latent space.
Zhu et al.
(2021) proposes to estimate label-label compatibility matrix and uses it as a prior to update posterior belief on the labels.
However, these approaches still rely on the given graphs for information like the hop neighbours etc., which might be poor approximations of the desired neighbours with similar labels.
Some of the recent approaches focused on learning ﬁlter functions that operate on the eigenvalues of the graph.
Learning these ﬁlter functions can be viewed as directly adapting the graph for the desired task.
Bo et al.
(2021) models the ﬁlter function as an attention mechanism on the edges, which learns the difference in the proportion of low-pass and high-pass frequency signals.
Chien et al.
(2021) proposes a polynomial ﬁlter on the eigenvalues that directly adapts the graph for the desired task.
Zheng et al.
(2021) decompose the graph into low-pass and high-pass frequencies, and deﬁne a framelet based convolutional model.
Our work is closely related to these lines of exploration.
In this work, we propose to learn a ﬁlter function as a sum of polynomials over different subsets of the eigenvalues, enabling design of effective ﬁlters to model task-speciﬁc complex frequency responses with compute trade-offs.
P ROBLEM S ETUP AND M OTIVATION
We focus on the problem of semi-supervised node classiﬁcation on a simple graph G =
( V , E ) , where V is the set of vertices and E is the set of edges.
Let A ∈ { 0 , 1 } n × n be the adjacency matrix associated with G , where n = |V| is the number of nodes.
Let Y be the set of all possible class labels.
Let X ∈ R n × d be the d -dimensional feature matrix for all the nodes in the graph.
Given a training set of nodes D ⊂ V whose labels are known, along with A and X , our goal is to predict the labels of the remaining nodes.
Let A I
=
A
+ I where I is the identity matrix.
Let D A
I be the degree matrix of A I and (cid:101)
A
= D − 1 / 2 A I A
I D − 1 / 2 A
I .
Let (cid:101)
A = UΛU T be the eigendecomposition of (cid:101) A .
Then, the spectral convolution of X on the graph A can be deﬁned via the reference operator (cid:101)
A and a ﬁlter function
h operating on the eigenvalues, in the Fourier domain (Tremblay et al., 2017; Chien et al., 2021) as,
[Equation]
where H ( Λ ) =
diag ( h ( Λ )) .
In this process, the ﬁlter function is essentially adapting the graph for the desired task at hand.
The second equality follows from using a polynomial ﬁlter,
H ( λ ) = (cid:80) ki
=
α
i λ i where α i ’s are coefﬁcients of the polynomial, k is the order of the polynomial and λ is any eigenvalue from Λ .
Note that H ( Λ ) =
diag ( V α ) where V ∈ R n × k is a Vandermonde matrix constructed using eigenvalues from Λ and α ∈ R k × 1 is the vector consisting of α j ’s.
The last equality follows from the eigendecomposition properties, enabling avoidance of eigendecomposition.
It is well-known that polynomial ﬁlters can approximate any graph ﬁlter (Shuman et al., 2013;
Tremblay et al., 2017).
However, several practical challenges arise in learning a good polynomial ﬁlter.
We brieﬂy discuss them below.
Let h ∗ be the optimal ﬁlter for some given task.
Learning a polynomial ﬁlter involves the approximation: h ∗ ≈ V α .
Notice that this system is over-determined since the number of unknowns ( k , the size of α ) is less than the number of equations ( n , the number of nodes).
To obtain a consistent solution, we may have to work with higher-order polynomials.
Working with higher powers poses several practical challenges, and the following observations are in order.
First, node features computed via (cid:101)
A j X becomes indistinguishable (see Figure 1a and A.2.1).
Since the node features are indistinguishable, the importance of the coefﬁcient associated with (cid:101)
A j X for the task at hand reduces signiﬁcantly.
These observations are in line with Theorem 4.2 in Chien et al.
(2021), where it states that as over-smoothing happens, the corresponding coefﬁcients of the polynomial go to zero.
We observe that with insigniﬁcant contributions from higher-order terms, the performance (aka test accuracy) does not improve with the increase in the order of the polynomial (See Figure 1b, 1c and A.2.2).
Next, one can think of addressing this issue with (cid:101)
A j X by directly working with full eigendecomposition.
However, this approach is prohibitively expensive for large graphs.
Even if we were to do this, λ i ’s are in the range of [ − 1 , 1] and thus λ ji will diminish with higher powers.
Therefore, it will continue to be less effective in learning coefﬁcients of a higher power.
Motivated by the above observations, we propose a novel piece-wise polynomial ﬁlter learning approach.
P ROPOSED A PPROACH
The central goal of our approach is to get improved task-speciﬁc performance by learning effective task-speciﬁc graph ﬁlters.
We propose to learn the polynomial ﬁlter as a sum of polynomials operating on different eigenvalue intervals, taking task-speciﬁc requirements and practical considerations into account.
We show that our proposed ﬁlter design can approximate the latent optimal graph
[Figure]
ﬁlter better than a single polynomial, and the resultant class of learnable ﬁlters/graphs is richer.
We brieﬂy discuss the training and computational complexity of the proposed model.
4. P IECE - WISE P OLYNOMIAL /S PLINE G RAPH F ILTER F UNCTION
We model the ﬁltering function h ( λ ) as a piece-wise polynomial or spline function where each polynomial is of a lower degree (e.g., a cubic polynomial).
We partition the eigen spectrum in [ − 1 , 1] (or [0 , 2] as needed) into contiguous intervals and approximate the desired frequency response by ﬁtting a low degree polynomial in each interval.
This process helps us to learn a more complex shaped frequency response as needed for the task.
Let S =
{ σ 1 , σ 2 , . . .
σ m } denote a partition set with m contiguous intervals and h i,k i ( λ ; γ i ) denote a k i -degree polynomial ﬁlter function of the interval σ i with polynomial coefﬁcients
γ i .
We deﬁne PP-GNN ﬁlter function as:
[Equation]
and learn a smooth ﬁlter function by imposing additional constraints to maintain continuity between polynomials of contiguous intervals at different endpoints ( aka knots).
This class of ﬁlter functions is rich, and its complexity is controlled by choosing intervals (i.e., endpoints and number of partitions) and polynomial degrees.
Given the ﬁlter function, we compute the node embedding matrix as:
[Equation]
where U i is a matrix with eigenvectors corresponding to eigenvalues that lie in σ i , H i ( γ i ) is the diagonal matrix with diagonals containing the h i evaluated at the eigenvalues and Z
( X ; Θ ) is an
MLP network with parameters Θ .
4. P RACTICAL AND I MPLEMENTATION C ONSIDERATIONS
The ﬁlter function (3) requires computing eigendecomposition of (cid:101) A and is expensive, therefore, not scalable for very large graphs.
Thus, having ﬁner control to learn complex frequency responses comes with a high computational cost.
We address this problem by devising a graph ﬁlter function that enables a trade-off between computational cost and frequency response control.
First, we make the following observations:
Top-k and bottom-k eigenvalues represent k low and high frequencies.
There are efﬁcient algorithms and off-the-shelf library packages available to get top-k and bottom-k eigenvalues and eigenvectors of sparse matrices.
GPR-GNN method also learns a graph ﬁlter but operates on the entire eigen spectrum by sharing the ﬁlter coefﬁcients across the spectrum.
Therefore, it is a special case of our proposed model (3).
Since GPR-GNN learns a global polynomial by having shared coefﬁcients across the spectrum (like learning a global polynomial to ﬁt a function), it is not ﬂexible enough to learn complex frequency responses, as needed in several practical applications.
Note that increasing the degree of the global polynomial does not help beyond some high degree speciﬁc to tasks.
We demonstrate this through extensive experiments on benchmark datasets in the experiment and appendix sections.
One important advantage of GPR-GNN is that it does not require computation of eigendecomposition; therefore, efﬁcient.
Many recent works, including GPR-GNN, investigated the problem of designing robust graph neural networks that work well across homophilic and heterophilic graphs.
They found that graph ﬁlters that amplify or attenuate low and high-frequency components of signals (i.e., low-pass and high-pass ﬁlters) are critical to improving performance on several benchmark datasets.
Efﬁcient Variant.
Using the above observations, we propose an efﬁcient variant of (2) as:
[Equation]
where S l consists of partitions over low frequency components, S h consists of partitions over high frequency components, the ﬁrst and second terms ﬁt piece-wise polynomials 1 in low and high frequency regions, as indicated through superscripts.
However, we do not want to lose any useful information from other frequencies in the central region, yet maintain efﬁciency.
We achieve this by adding the GPR-GNN ﬁlter function, h gpr ( λ ; γ ) , which is computationally efﬁcient.
Since h gpr ( λ ; γ ) is a special case of (2) and the terms in (4) are additive, it is easy to see that (4) is same as (2) with a modiﬁed set of polynomial coefﬁcients.
In practice, the required low and high-frequency components are computed based on affordable computational cost.
Furthermore, we can also control the contributions from each term by setting or optimizing over hyperparameters, η l , η h and η gpr .
Thus, the proposed model offers richer capability and ﬂexibility to learn complex frequency response and balance computation costs over GPR-GNN.
Model Training.
Like GPR-GNN, we apply S OFTMAX activation function on (3) and use the standard cross-entropy loss function to learn the sets of polynomial coefﬁcients ( γ ) and classiﬁer model parameters ( Θ ) using labeled data.
To ensure smoothness of the learned ﬁlter functions, we add a regularization term that penalizes squared differences between the function values of polynomials of contiguous intervals at each other’s interval end-points.
More details can be found in the appendix (A.4.1).
Computational Cost.
There is some pre-training cost of computing the eigendecomposition for top and bottom k eigenvalues.
Most algorithms for this task utilize Lanczos’ iteration, convergence bounds of which depends on the input matrix’ spectrum (Saad, 1980; LI, 2010), which although have superlinear convergence, but are observed to be efﬁcient in practice.
We compute node embeddings afresh whenever the model parameters are updated.
This computation involves matrix multiplication with eigenmatrices incurring an additional cost (over GPR-GNN) of O ( nkL ) where k and L denote the number of selected low/high eigenvalues and classes, respectively.
4. A NALYSIS
We ﬁrst give a simple theoretical justiﬁcation for using multiple polynomials.
We upper-bound the approximation error achieved by using multiple polynomials by the error that the single polynomial parameterization achieves.
We relegate the proofs of all the theorems of this section to the appendix.
Theorem 4.1.
For any frequency response h ∗ , and an integer K ∈ N , let ˜ h :
= h + h f , with h f having a continuous support over a subset of the spectrum, σ f . Assume that h and h f are parameterized by independent K and K (cid:48) -order polynomials, p and p f , respectively, with K (cid:48) ≤ K .
Then there exists ˜ h , such that min (cid:107) ˜
h − h ∗ (cid:107) 2 ≤ min (cid:107) h − h ∗ (cid:107) 2 , where the minimum is taken over the polynomial parameterizations.
Moreover, for multiple polynomial adaptive ﬁlters h f 1 , h f 2 , ..., h f m 1 For brevity, we dropped the polynomial degree dependency.
parameterized by independent K (cid:48) -degree polynomials with K (cid:48)
≤ K but having disjoint, contiguous supports, the same inequality holds for ˜ h
= h + (cid:80) mi
=1 h f i .
For a detailed proof please refer to A.4.3.
We constructed waveforms of arbitrary complexity, approximated by ﬁrst ﬁtting a single globally-deﬁned polynomial and then adding locally deﬁned polynomials on top of it.
We assess the performance by varying the degrees of the local polynomials.
The corresponding plots (See Figure 8) indicate that use of lower-order ﬁlters achieves a ﬁt comparable to one by a relatively higher-degree polynomial.
Since an actual waveform is not observed in practice and instead, we estimate it by optimizing over the observed labels via learning a graph ﬁlter, we theoretically show that the family of ﬁlters that we learn is a strict superset of the polynomial ﬁlter family.
The same result holds for the families of the resulting adapted graphs.
Theorem 4.2.
Deﬁne H :
= { h ( · ) | ∀ possible K-degree polynomial parameterizations of h } to be the set of all K-degree polynomial ﬁlters, whose arguments are n × n diagonal matrices, such that a ﬁlter response over some Λ is given by h ( Λ ) for h ( · ) ∈ H .
Similarly H (cid:48) :
= { ˜ h ( · ) | ∀ possible polynomial parameterizations of ˜ h } is set of all ﬁlters learn-able via PP-GNN , with ˜ h
= h
+ h f
+ h f 2 , where h is parameterized by a K-degree polynomial supported over entire spectrum, h f 1 and h f 2 are adaptive ﬁlters parameterized by independent K (cid:48)
-degree polynomials which only act on top and bottom t diagonal elements respectively, with t < n/ 2 and K (cid:48) ≤ K ; then
H and H (cid:48) form a vector space, with H ⊂ H (cid:48) .
Also, dim ( H (cid:48) ) dim ( H ) =
K +2 K (cid:48)
+3 K + .
Corollary 4.2.1.
The corresponding adapted graph families G :=
{ U T h ( · ) U T | ∀ h ( · ) ∈ H } and
G (cid:48) :
= { U T ˜ h ( · ) U T | ∀ ˜ h ( · ) ∈ H (cid:48) } for any unitary matrix U form a vector space, with G ⊂ G (cid:48) and dim ( G (cid:48) )
dim ( G ) =
K +2 K (cid:48)
+3 K + .
This implies that our model is learning from a more diverse space of ﬁlters and the corresponding adapted graphs.
Moreover, the dimension of the space increases signiﬁcantly by using just two adaptive ﬁlters.
Note that learning from such a diverse region is feasible.
This observation is possible from the proofs of Theorem 4.2 (A.4.4) and Corollary 4.2.1 (A.4.5).
Using the adaptive ﬁlters without any ﬁlter with the entire spectrum as support results in learning a set of adapted graphs, ˆ G .
This set is disjoint from G , with G (cid:48)
=
G ⊕ ˆ G .
We conduct various ablative studies where we demonstrate the effectiveness of learning from ˆ G and G (cid:48) .
E XPERIMENTS
We conduct comprehensive experiments to demonstrate the effectiveness and competitiveness of the proposed PP-GNN model over state-of-the-art (SOTA) models/methods and answer the following research questions.
[RQ1] Does the PP-GNN model outperform SOTA models/methods over a broad range of datasets (homophilic/heterophilic)?
[RQ2] Does the proposed model learn better and complex frequency responses that improve performance [Frequency response plots]?
[RQ3] With the PP-GNN model comprising of multiple ﬁlters, how does each ﬁlter contribute to achieving improved performance?
[RQ4 and RQ5] Does the model learn better embedding, and do we need a large number of eigenvalues/vectors and polynomials to get improved performance?
[RQ6] How does the training time of
the PP-GNN and GPR-GNN models compare?
Datasets and Setup.
We evaluate our model on several real-world heterophilic and homophilic datasets including a few large graphs for the node classiﬁcation task.
Detailed statistics of the benchmark datasets are provided in the appendix (A.5.1).
The heterophilic datasets include Texas 2 ,
Cornell 2 , Wisconsin 2 , Chameleon , Squirrel , (Rozemberczki et al., 2021) and Flickr .
We use the 10 random splits (48%/32%/20% of nodes for train/validation/test set) from Pei et al.
For Flickr , Ogbn-Arxiv , and Wiki-CS , we use the data splits from Kim & Oh (2021).
We also evaluate on 8 homophilic datasets: Cora-Full , Ogbn-Arxiv , Wiki-CS , Citeseer , Pubmed , Cora ,
Computer , and Photos borrowed from Kim & Oh (2021).
For the remaining homophilic datasets, we create 10 random splits for each dataset following Kim & Oh (2021).
We use PCA node features (Kim & Oh (2021)) for the Chameleon and Cora-Full datasets.
We report mean and standard deviation of test accuracy over splits to compare model performance.
Baselines.
We compare our model against (a) ﬁltering based approaches, GPR-GNN, FAGCN,
APPNP, and LGC, (b) other state-of-the-art models include S UPER GAT, TDGNN, H 2 GCN, and G EOM -GCN, and (c) standard baselines, namely, LR (Logistic Regression), MLP (Multi Layer
Perceptron), GCN, and SGCN.
Detailed descriptions for all the baselines, hardware and software speciﬁcations are in the appendix (A.5.2 and A.5.5)
Hyperparameter Tuning.
For the PP-GNN model, we separately partition the low-end and highend eigenvalues into several contiguous partitions and use shared ﬁlter parameters for frequencies of each partition.
The number of partitions, which can be interpreted as the number of ﬁlters, is swept in the range [2,3,4,5,10,20].
The polynomial ﬁlter order is swept in the range [1,10] in steps of size 1.
The number of eigenvalues/vectors are swept in the range [32, 64, 128, 256, 512, 1024].
In our experiments, we set η l = η h and we vary them in range (0, 1) and set η gpr = 1 − η l .
We did hyperparameter tuning for other models as suggested in respective references and public repositories.
More details are in the appendix (A.5.5).
[Table] [Table]
5. RQ1: PP-GNN VERSUS SOTA M ODELS
We present several important observations from Tables 1 and 2.
We observe that the PP-GNN model consistently outperforms all models, including recent ﬁltering approach based models on all datasets (both heterophilic and homophilic) except Pubmed and Cornell, where it achieves similar performance.
This result demonstrates the effectiveness and robustness of our model across a wide variety of datasets.
Furthermore, compared with the GPR-GNN model, learning piece-wise polynomial ﬁlters improves performance signiﬁcantly over learning a single polynomial ﬁlter.
In particular, PP-GNN achieves performance improvements of around 10% and 5% on the Chameleon and Squirrel datasets.
5. RQ2: A DAPTABLE F REQUENCY R ESPONSES
We computed the frequency responses (i.e., h ( λ ) ) of learned polynomials of the PP-GNN and GPRGNN models on several datasets, including Squirrel and Citeseer datasets shown in Figure 2.
We observe from Figure 2a, though the GPR-GNN model can learn some variations at low/high frequencies, it is insufﬁcient to achieve higher classiﬁcation accuracy.
On the other hand, the PPGNN model can capture complex shapes at the low and high ends of the spectrum, enabling it to achieve signiﬁcantly improved test accuracy.
We observed a similar phenomenon for the chameleon dataset as well.
To illustrate another behaviour, we present the frequency responses for the Citeseer dataset in Figure 2b, and the responses are similar except for some variations at the low-end of the spectrum.
Note that the GPR-GNN model does well on several homophilic datasets.
These observations suggest that the PP-GNN model adapts very well in learning desired frequency responses, as dictated by the task at hand.
We can observe such a behaviour for two other datasets in the appendix (A.5.6).
[Figure]
and GPR-GNN 5. RQ3: P ERFORMANCE COMPARISON OF D IFFERENT F ILTERS
Recall that the PP-GNN model is a sum of polynomials model comprising of polynomial ﬁlters operating at different parts of the spectrum.
Here, we study the importance and effect of using a combination of ﬁlters operating on different regions.
For reference, we also compare the performance of GPR-GNN model that operates on the entire spectrum using a single ﬁlter.
Several interesting observations are in order.
From Table 3, we see that the best performance is achievable using high-frequency signals alone for the heterophilic datasets (Squirrel and Chameleon), suggesting that signiﬁcant discriminatory information is available at high frequencies compared to low-frequency signals.
In contrast, homophilic datasets (Cora and Citeseer) exhibit a reverse trend.
On comparing the ﬁrst (second) and third (fourth) row results, we see that having the GPR-GNN ﬁlter as part of the PP-GNN ﬁlter helps to get improved performance over individual ﬁlters (PP-GNN(Low) or PP-GNN (High)).
Finally, the PP-GNN model can adapt well, capture contrasting information bands across datasets, and outperform the GPR-GNN model.
[Figure]
[Table]
5. RQ4 - RQ6: V ARYING N O . OF EV S , L EARNED EMBEDDING AND T IMING C OMPARISON
We conducted ablative studies to see the effect of varying different hyperparameters of the PP-GNN model.
From Figure 3a
[Table]
we see that as small as 32 eigen components are sufﬁcient to achieve nearbest performance on the homophilic datasets (Cora, Citeseer).
On the other hand, peak performance is achieved for some intermediate ( ≈ 250-500) number of eigen components for the Squirrel and
Chameleon datasets.
Also, we note that we can achieve comparable performance to state of the art models on large datasets (Flickr and OGBN-Arxiv) using only 1024 components (See Tables 1 and 2).
This study suggests that the PP-GNN model is scalable at least for medium sized graphs ( 100K nodes).
More evaluation and investigation are needed to study performance versus computation cost trade-off for very large graphs (millions of nodes).
To assess the quality of learned embedding, we also created t-SNE plots and made a visual inspection.
From Figure 3b and 3c, we see that the clusters belonging to different classes of the Squirrel dataset are better separated for the PP-GNN model compared to the GPR-GNN model.
We also measured the time taken by the GPR-GNN and PP-GNN models.
We observed that the GPRGNN model is only (1 . 2 − 2) × faster than the PP-GNN model (using 1024 eigencomponents) on many datasets.
Thus, the extra training cost incurred by the PP-GNN model is not signiﬁcantly different for practical purposes.
More details are in the appendix (A.6).
C ONCLUSION
Several proposed models attempt to be robust to the correlations between graph and node labels.
We build on the ﬁlter-based approach of GPR-GNN (Chien et al., 2021).
This work proposed an effective polynomial ﬁlter design.
We combine GPR-GNN with additional polynomials that adapt speciﬁcally to low and high-frequency components.
Our experiments demonstrate that such an approach can learn ﬁlter functions that improve performance on the node classiﬁcation task.
Our plots of these ﬁlter functions suggest that they are of high order on several datasets.
It would be interesting to analyze these ﬁlter plots and identify some common characteristics.
These will enable us to a] characterize the correlation of graphs and labels, b] further improve the performance and c] build robust graph privacy models.
We plan to do this as our future work.
R EPRODUCIBILITY S TATEMENT
We have taken several steps to ensure the reproducibility of our work.
First, we provide a detailed description of our model in Section 4.
Section 5 provides a brief description of the datasets utilized in our experiments.
A detailed description of the datasets, including their sources, statistics are given in the appendix (A.5.1).
We also provide details on the number of splits and the method used to generate them.
We also list out all the baselines utilized in the experiments in Section 5.
A detailed description of these baselines and the parameter sweep ranges are given in the appendix (A.5.2).
We also provide implementation detail of our model in the appendix (A.5.5).
Additionally, we share an anonymized URL of the code for our model, and the data splits in the appendix (A.1).
E THICS S TATEMENT
Social network graphs form the most exciting application for GNNs.
GNNs can be used to reveal user information that the user otherwise would have preferred to keep private.
Privacy-preserving methods might rely on obfuscating the graph by adding spurious edges, and these edges will, in turn, reduce the graph’s correlation with the node labels.
Earlier GNNs would likely not have predicted the target label with reasonable accuracy under this setting.
Thereby, such obfuscation methods would give some level of privacy for the users.
However, our model exhibits good performance even in the presence of a lower correlation.
Such methods can make simple obfuscation based privacy approaches obsolete while still revealing important information about the users.
On the other hand, our model can also give insights into what makes graphs reveal certain information and develop a more robust privacy model for graphs.
A A PPENDIX
The appendix is structured as follows.
In Section A.1, we provide a URL with our code and data, along with some information about the contents in the URL.
In Section A.2, we present additional evidence of the limitations of GPR-GNN.
In Section A.3, we show a representative experiment that motivates Section 4.
In Section A.4, we provide proofs for theorems and corollaries deﬁned in Section 4.3.
In Section A.5, we provide more details regarding the baselines, datasets and their respective splits.
We also provide implementation details and rank PP-GNN against current SoTA.
In Section A.6, we perform a timing analysis where we compare PP-GNN to GPR-GNN.
A. C ODE AND D ATASET
Our code, along with the datasets and their respective splits, is available at https://tinyurl.
com/PPGNN .
A README ﬁle is available explaining how the code can be executed on various datasets.
The requirements.txt ﬁle lists all the necessary python packages required to execute the code.
The src folder also contains bash scripts to run the code on various datasets.
A. M OTIVATION A.2. N ODE F EATURE I NDISTINGUISHABLY P LOTS
In the main paper (Figure 1a), we plot the average of pairwise distances between node features for the Cora dataset, after computing (cid:101)
A j X for increasing j values, and showed that the mean pairwise node feature distance decreases as j increases.
We observe that this is consistent across three more datasets: Citeseer, Chameleon and Squirrel.
This is observed in Figure 4.
[Figure]
We also observed the mean of the variance of each dimension of node features, after computing (cid:101)
A j X , for increasing j values.
We observe that this mean does indeed reduce as the number of hops increase.
We also observe that the variance of each dimension of node features reduces for Cora,
Squirrel and Chameleon as the number of hops increase; however, we don’t observe such an explicit phenomenon for Citeseer.
See Figure 5.
A.2. E FFECT OF VARYING THE ORDER OF THE GPR-GNN P OLYNOMIAL
In the main paper (Figure 1a), we plot the test accuracies of the GPR-GNN model while increasing the order of the polynomials for the Cora and Chameleon dataset, respectively.
We observe that on increasing the polynomial order, the accuracies do not increase any further.
We can show a similar phenomenon on two other datasets, Squirrel and Citeseer, in Figure 6.
In Section 3 of the main paper, we claim that due to the over-smoothing effect, even on increasing the order of the polynomial, there is no improvement in the test accuracy.
Moreover, in Figurewe can see that our model can learn a complicated ﬁlter polynomial while GPR-GNN cannot.
This section shows that even on increasing the order of the GPR-GNNpolynomial, neither does the test accuracy increase nor does the waveform become as complicated as PP-GNN.
See Figure 7.
[Figure] [Figure]
A. F ICTITIOUS P OLYNOMIAL
In Section 4, we claim that having multiple disjoint low order polynomials can approximate a complicated waveform better than a higher-order polynomial.
We create a representative experiment that shows this is indeed true by creating a ﬁctitious complicated polynomial and trying to ﬁt it using both a single polynomial and the disjoint multi polynomial.
We observe in Figure 8 that the lower order disjoint polynomial ﬁts the complicated waveform better than a higher-order single polynomial.
A. P ROPOSED A PPROACH A.4. D ETAILS REGARDING BOUNDARY REGULARIZATION
To induce smoothness in the learned ﬁlters, we add a regularization term that penalizes squared differences between function values of polynomials at knots (endpoints of contiguous bins).
Our regularization term looks as follows:
[Figure] [Figure]
[Equation]
In equation 5, σ maxi and σ mini refer to the maximum and minimum eigenvalues in σ i (Refer to
Section 4).
This regularization term is added to the Cross-Entropy loss.
We perform experiments with this model and report the performance in Table 4.
We observe that we are able to reach similar performance even without the presence of this regularization term.
Therefore, majority of the results reported in our Main paper are without this regularization term.
A.4. N OTATION U SED
Vectors are denoted by lower case bold Roman letters such as x , and all vectors are assumed to be column vectors.
In the paper, h with any sub/super-script refers to a frequency response, which is
[Table]
also considered to be a vector.
A superscript T denotes the transpose of a matrix or vector; Matrices are denoted by bold Roman upper case letters, such as M .
A ﬁeld is represented by K ; sets of real and complex numbers are denoted by R and C respectively.
K [ x 1 , . . .
, x n ] denotes a multivariate polynomial ring over the ﬁeld K , in indeterminates x 1 , . . .
, x n .
Set of n × n square matrices with entries from some set S are denoted by M n ( S ) .
Moore-Penrose pseudoinverse of a matrix A is denoted by A † .
Eigenvalues of a matrix are denoted by λ , with λ 1 , λ 2 , . . .
denoting a decreasing order when the eigenvalues are real.
A matrix Λ denotes a diagonal matrix of eigenvalues.
Set of all eigenvalues, i.e., spectrum, of a matrix is denoted by σ A or simply σ when the context is clear.
L p norms are denoted by (cid:107) · (cid:107) p .
Frobenius norm over matrices is denoted by (cid:107) · (cid:107) F .
Norms without a subscript default to L 2 norms for vector arguments and Frobenius norm for matrices.
⊕ denotes a direct sum.
For maps f i deﬁned from the vector spaces V 1 , · · · , V m , with a map of the form f :
V (cid:55)→ W , with V =
V 1 ⊕ V 2 ⊕· · ·⊕ V m , the the phrase ” f : V (cid:55)→ W by mapping f ( v i ) to f i ( g ( v i )) ” means that f maps a vector v = v 1 + . . .
+ v m with v i ∈ V i to f 1 ( g ( v 1 )) + . . .
+ f m ( g ( v m ))
A.4. P ROOF OF T HEOREM 4.
Theorem.
For any desired frequency response h ∗ , and an integer K ∈ N , let ˜ h :
= h + h f , with h f having a continuous support over a subset of the spectrum, σ f .
Assuming h and h f to be parameterized by independent K and K (cid:48) -order polynomials p and p f respectively, with K (cid:48) ≤ K , then there exists ˜ h , such that min (cid:107) ˜
h − h ∗ (cid:107) 2 ≤ min (cid:107) h − h ∗ (cid:107) 2 , where the minimum is taken over the polynomial parameterizations.
Moreover, for multiple polynomial adaptive ﬁlters h f 1 , h f 2 , ..., h f m parameterized by independent K (cid:48) -degree polynomials with K (cid:48)
≤ K but having disjoint, contiguous supports, the same inequality holds for ˜ h
= h + (cid:80) mi
=1 h f i .
Proof.
We make the following simplifying assumptions: 1.
| σ f i | > K, ∀ i ∈ [ m ] , i.e., that is all support sizes are lower bounded by K (and hence K (cid:48) ) 2.
All eigenvalues of the reference matrix are distinct
For methods that use a single polynomial ﬁlter, the polynomial graph ﬁlter, h K ( Λ ) =
diag ( V γ ) where γ is a vector of coefﬁcients (i.e, γ parameterizes h ), with eigenvalues sorted in descending order in components, and V is a Vandermonde matrix:
[Equation]

[Equation]
· · · λ Kn  
And to approximate a frequency response h ∗ , we have the following objective:
[Equation]
Where (cid:107)(cid:107) F and (cid:107)(cid:107) 2 are the Frobenius and L 2 norms respectively.
Due to the assumptions, the system of equations h ∗ =
V γ is well-deﬁned and has a unique minimizer, γ ∗ =
V † h ∗ , and thus
[Equation]
e p
[Equation]
γ ∗ )
[Equation]
= min γ (cid:107) e p ( γ ) (cid:107) .
Next we break this error vector as:
[Equation]
Where e ∗ p
i :=
( h ∗ i − V
i γ ∗ ) with similar deﬁnition for e p L ; h ∗ i is a vector whose value at components corresponding to the set σ ( h f i ) is same as that of h ∗ and rest are zero.
Similarly, V ∗ i is a matrix whose rows corresponding to the set σ ( h f i ) are same as that of V with other rows being zero.
Also, V L = V − (cid:80) mi =
V i and h ∗ L = h ∗ − (cid:80) mi =
h ∗ i .
Note that as a result of this construction, [ e p ∗ i ] ∪ e p ∗ L is a linearly independent set since the supports [ σ ( h f i ) ] form a disjoint set (note the theorem statement).
We split the proof in two cases:
Case 1: K (cid:48)
=
K .
We now analyze the case where we have m polynomial adaptive ﬁlters added, all having an order of K , where the objective is min (cid:107) ˜
h − h ∗ (cid:107) , which can be written as:
[Equation]
Before characterizing the above system, we break a general error vector as:
[Equation]
Where e g i :=
( h ∗ i − V i ( γ + γ i )) with similar deﬁnition for e g L .
Clearly, the systems of equations, e g i = 0 , ∀ i and e g L = 0 are well-deﬁned due to the assumptions 1 and 2.
Since all the systems of equations have independent argument, unlike in the polynomial ﬁlter case where the optimization is constrained over a single variable; one can now resort to individual minimization of squared norms of e g i which results in a minimum squared norm of e g .
Thus, we can set:
[Equation]
to minimize squared norms of e g
i and e g L .
Note that [ e g i ] ∪ e g L is a linearly independent set since the supports [ σ ( h f i ) ] form a disjoint set and by the above construction, this is also an orthogonal set, and hence we have (cid:107) e g (cid:107)
= (cid:80) mi =1 (cid:107)
e g i (cid:107)
+ (cid:107) e g L (cid:107) 2 , and hence the above assignment implies:
[Equation]
e g ( γ ∗ g , [ γ ∗ i ]) (cid:107) =
min γ, [ γ i ] (cid:107) e g ( γ g , [
[Equation]
i ]) (cid:107) :
= min (cid:107) ˜
h
[Equation]
h ∗ (cid:107)
Hence, it follows that, min x (cid:107) h ∗
i − V
i x (cid:107) 2 = (cid:13)(cid:13)
e ∗ g
i (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)
e ∗ p
i (cid:13)(cid:13)
= (cid:107) h ∗
i − V
i γ ∗ (cid:107) 2 and min x (cid:107) h ∗ L −
V L x (cid:107) 2 = (cid:13)(cid:13)
e ∗ g L (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13)
e ∗ p L (cid:13)(cid:13)
= (cid:107) h ∗ L − V L γ ∗ (cid:107) 2 .
Hence, m (cid:88) i =
(cid:13)(cid:13)
e ∗ g
i (cid:13)(cid:13) 2 + (cid:13)(cid:13)
e ∗ g L (cid:13)(cid:13) 2 ≤ m (cid:88) i =
(cid:13)(cid:13)
e ∗ p
i (cid:13)(cid:13) 2 + (cid:13)(cid:13)
e ∗ p L (cid:13)(cid:13)min (cid:107) ˜
h − h ∗ (cid:107) ≤ min (cid:107) h − h ∗ (cid:107)
Case 2: K (cid:48) <
K .
We demonstrate the inequality showing the existence of an ˜ h that achieves a better approximation error.
By deﬁnition, the minimum error too will be bounded above by this error.
For this, we ﬁx γ , the parameterization of h as γ = V † h ∗ =
γ ∗ p ( say ) .
Note that γ p ∗ = arg min γ (cid:107) e p ( γ ) (cid:107) .
Now our objective function becomes
[Equation]
Where h ∗ i , h ∗ L , V i , V L have same deﬁnitions as that in case 1 and V (cid:48)
i is a matrix containing ﬁrst
K (cid:48)
+ 1 columns of V i as its columns (and hence has full column rank), and, γ i ∈ R K (cid:48)
+1 .
By this construction, we have
[Equation]
Our optimization objective becomes min [
[Equation]
i ]
[Equation]
e g
[Equation]
i ]) (cid:107) , which is easy since the problem is wellposed by assumption 1 and 2.
The unique minimizer of this is obtained by setting
[Equation]
Now,
[Equation]
= min γ (cid:107) e p ( γ ) (cid:107) = min (cid:107) h − h ∗ (cid:107)
By the deﬁnition of minima, min γ, [ γ i ] (cid:107) e g ( γ, [ γ i ]) (cid:107) ≤ min [ γ i ] (cid:107) e g ( γ ∗ p , [ γ i ]) (cid:107) , and by the deﬁnition, min (cid:107) ˜
h − h ∗ (cid:107) =
min γ, [ γ i ] (cid:107) e g ( γ, [ γ i ]) (cid:107) , we have:
[Equation]
A.4. P ROOF OF T HEOREM 4.
Theorem.
Deﬁne H :
= { h ( · ) | ∀ possible K-degree polynomial parameterizations of h } to be the set of all K-degree polynomial ﬁlters, whose arguments are n × n diagonal matrices, such that a ﬁlter response over some Λ is given by h ( Λ ) for h ( · ) ∈ H .
Similarly H (cid:48) :
= { ˜ h ( · ) | ∀ possible polynomial parameterizations of ˜ h } is set of all ﬁlters learn-able via PP-GNN , with ˜ h
= h
+ h f
+ h f 2 , where h is parameterized by a K-degree polynomial supported over entire spectrum, h f 1 and h f 2 are adaptive ﬁlters parameterized by independent K (cid:48)
-degree polynomials which only act on top and bottom t diagonal elements respectively, with t < n/ 2 and K (cid:48) ≤ K ; then
H and H (cid:48) form a vector space, with H ⊂ H (cid:48) .
Also, dim ( H (cid:48) ) dim ( H ) =
K +2 K (cid:48)
+3 K + .
Proof.
We start by constructing the abstract spaces on top of the polynomial vector space.
Consider the set of all the univariate polynomials having degree at most K in the vector space over the ring K xn :
=
K [ x 1 , . . .
, x n ] where K is the ﬁeld of real numbers.
Partition this set into n subsets, say
V 1 , . .
. , V n , such that for i ∈ [ n ] , V i contains all polynomials of degree up to K in x i .
It is easy to see that V 1 , . .
, V n are subspaces of K [ x 1 , . . .
, x n ] .
Deﬁne V =
V 1 ⊕ V 2 ⊕ · · · ⊕ V n where ⊕ denotes a direct sum.
Deﬁne the matrix D
i
[ c ] whose ( i, i ) th entry is c and all the other entries are zero.
For i ∈ [ n ] , deﬁne linear maps φ i : V i → M n ( K xn ) by f ( x i )
(cid:55)→ D i [ f ( x i )] .
Im( φ i ) forms a vector space of all diagonal matrices, whose ( i, i ) entry is the an element of V i .
Generate a linear map φ : V → M n ( K xn ) by mapping φ ( f ( x i )) to φ i ( f ( x i )) for all i ∈ [ n ] as the components of the direct sum present in its argument.
Note that φ i for i ∈ [ n ] are injective maps, making φ an injective map.
This implies that H ⊂ Im ( φ ) is a subspace with basis B h :=
{ φ ( x 0 1 + · · · + x 0 n ) , φ ( x 1 + · · · + x n ) , . . .
, φ ( x K 1 + · · · + x Kn ) } , making dim ( H ) =
K +1 .
Similarly we have, H (cid:48) ⊂
Im ( φ ) , a subspace with basis B h (cid:48) :
= B h (cid:83) { φ ( x 01 + · · · + x 0 t +0+ · · · +0) , φ ( x 1 + · · · + x t +0+ · · · +0) , . . .
, φ ( x K (cid:48) 1 + · · · + x K (cid:48) t + 0 + · · · + 0) } (cid:83) { φ (0 + · · · + 0 + x 0 n − t +1 + · · · + x 0 n ) , φ (0 + · · · + 0 + x n − t +1 + · · · + x n ) , . . .
, φ (0+ · · · +0+ x K (cid:48) n − t +1 + · · · + x K (cid:48) n ) } where x 0 i and 0 are the corresponding multiplicative and additive identities of K x n , implying H ⊂ H (cid:48) and dim ( H (cid:48) )
=
K + 2 K (cid:48)
+ 1 .
A.4. P ROOF OF C OROLLARY 4.2.
Corollary.
The corresponding adapted graph families G :=
{ U T h ( · ) U T | ∀ h ( · ) ∈ H } and G (cid:48) :
= { U T ˜ h ( · ) U T | ∀ ˜ h ( · ) ∈ H (cid:48) } for any unitary matrix U form a vector space, with G ⊂ G (cid:48) and dim ( G (cid:48) )
dim ( G ) =
K +2 K (cid:48)
+3 K + .
Proof.
Consider the injective linear maps f 1 , f 2 : M n ( K xn )
→ M n ( K xn ) as f 1 ( A ) =
U T A and f 2 ( A ) =
AU .
Deﬁne f 3 : H → M n ( K xn ) and f 4 : H (cid:48)
→ M n ( K xn ) as f 3 ( A ) =
( f 1 ◦ f 2 )( A ) for
A ∈ H and f 4 ( A ) =
( f 1 ◦ f 2 )( A ) for A ∈ H (cid:48) .
Since U is given to be a unitary matrix, f 3 and fare monomorphisms.
Using this with the result from Theorem 4.2, H ⊂ H (cid:48) , we have G ⊂ G (cid:48) .
A. E XPERIMENTS A.5. D ATASETS
We evaluate on multiple benchmark datasets to show the effectiveness of our approach.
Detailed statistics of the datasets used are provided in Table 5.
We borrowed Texas , Cornell , Wisconsin from WebKB 3 , where nodes represent web pages and edges denote hyperlinks between them.
Actor is a co-occurrence network borrowed from Tang et al.
(2009), where nodes correspond to an actor, and and edge represents the co-occurrence on the same Wikipedia page.
Chameleon , Squirrel are borrowed from Rozemberczki et al.
Nodes correspond to web pages and edges capture mutual links between pages.
For all benchmark datasets, we use feature vectors, class labels from Kim & Oh (2021).
For datasets in (Texas, Wisconsin, Cornell, Chameleon, Squirrel, Actor), we use 10 random splits (48%/32%/20% of nodes for train/validation/test set) from Pei et al.
We borrowed Cora , Citeseer , and Pubmed datasets and the corresponding train/val/test set splits from Pei et al.
The remaining datasets were borrowed from Kim & Oh (2021).
We follow the same dataset setup mentioned in Kim & Oh (2021) to create 10 random splits for each of these datasets.
We also experiment with two slightly larger datasets Flickr Chua et al.
(July 8-10, 2009) and
OGBN-arXiv Hu et al.
We use the publicly available splits for these datasets.
[Figure]
[Table]
A.5. B ASELINES
We provide the methods in comparison along with the hyper-parameters ranges for each model.
For all the models, we sweep the common hyper-parameters in the same ranges.
Learning rate is swept over [0 . 001 , 0 . 003 , 0 .
01] , dropout over [0 . 2 , 0 .
weight decay over [1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 , 1 e − 2 , 5 e − 2 , 1 e − 1] , and hidden dimensions over [16 , 32 , 64] .
For model-speciﬁc hyper-parameters, we tune over author prescribed ranges.
We use undirected graphs with symmetric normalization for all graph networks in comparison.
For all models, we report test accuracy for the conﬁguration that achieves the highest validation accuracy.
We report standard deviation wherever applicable.
LR and MLP: We trained Logistic Regression classiﬁer and Multi Layer Perceptron on the given node features.
For MLP, we limit the number of hidden layers to one.
GCN: We use the GCN implementation provided by the authors of Chien et al.
Note: We observe discrepancies in performance with various implementations of GCN.
We make a note of this in Section A.7.
3 http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-11/www/wwkb
SGCN: SGCN (Wu et al., 2019) is a spectral method that models a low pass ﬁlter and uses a linear classiﬁer.
The number of layers in SGCN is treated as a hyper-parameter and swept over [1 , 2] .
S UPER GAT: S UPER GAT (Kim & Oh, 2021) is an improved graph attention model designed to also work with noisy graphs.
S UPER GAT employs a link-prediction based self-supervised task to learn attention on edges.
As suggested by the authors, on datasets with homophily levels lower than 0.
we use S UPER GAT SD .
For other datasets, we use S UPER GAT MX .
We rely on authors code 4 for our experiments.
G EOM -GCN: G EOM -GCN (Pei et al., 2020) proposes a geometric aggregation scheme that can capture structural information of nodes in neighborhoods and also capture long range dependencies.
We quote author reported numbers for Geom-GCN.
We could not run Geom-GCN on other benchmark datasets because of the unavailability of a pre-processing function that is not publicly available.
H 2 GCN: H 2 GCN (Zhu et al., 2020) proposes an architecture, specially for heterophilic settings, that incorporates three design choices: i) ego and neighbor-embedding separation, higher-order neighborhoods, and combining intermediate representations.
We quote author reported numbers where available, and sweep over author prescribed hyper-parameters for reporting results on the rest datasets.
We rely on author’s code 5 for our experiments.
FAGCN: FAGCN (Bo et al., 2021) adaptively aggregates different low-frequency and highfrequency signals from neighbors belonging to same and different classes to learn better node representations.
We rely on author’s code 6 for our experiments.
APPNP:
APPNP (Klicpera et al., 2019) is an improved message propagation scheme derived from personalized PageRank.
APPNP’s addition of probability of teleporting back to root node permits it to use more propagation steps without oversmoothing.
We use GPR-GNN’s implementation of
APPNP for our experiments.
L IGHT GCN: L IGHT GCN (Navarin et al., 2020) is a spectrally grounded GCN that adapts the entire eigen spectrum of the graph to obtain better node feature representations.
GPR-GNN: GPR-GNN (Chien et al., 2021) adaptively learns weights to jointly optimize node representations and the level of information to be extracted from graph topology.
We rely on author’s code 7 for our experiments.
TDGNN: TDGNN (Wang & Derr, 2021) is a tree decomposition method which mitigates feature smoothening and disentangles neighbourhoods in different layers.
We rely on author’s code 8 for our experiments.
ARMA:
ARMA (Bianchi et al., 2021) is a spectral method that uses K stacks of ARMA 1 ﬁlters in order to create an ARMA K ﬁlter (an ARMA ﬁlter of order K ).
Since (Bianchi et al., 2021) do not specify a hyperparameter range in their work, following are the ranges we have followed:
GCS stacks ( S ): [1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10] , stacks’ depth( T ): [1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10] .
However we only select conﬁgurations such that the number of learnable parameters are less than or equal to those in PP-GNN.
The input to the ARMAConv layer are the node features and the output is the number of classes.
This output is then passed through a softmax layer.
We use the implementation from the ofﬁcial PyTorch Geometric Library
BernNet: BernNet (He et al., 2021) is a method that approximates any ﬁlter over the normalised
Laplacian spectrum of a graph, by a K th Order Bernstein Polynomial Approximation.
We use the model speciﬁc hyper-parameters prescribed by the authors of the paper.
We vary the Propagation
Layer Learning Rate as follows:
We also vary the Propagation Layer 4 https:/
Dropout as follows: [0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8] .
We rely on the authors code 10 for our experiments.
AdaGNN: AdaGNN (Dong et al., 2021) is a method that captures the different importance’s for varying frequency components for node representation learning.
We use the model speciﬁc hyperparameters prescribed by the authors of the paper.
The No. of Layers hyper-parameter is varied as follows: [2 , 4 , 8 , 16 , 32 , 128] .
We rely on the authors code 11 for our experiments.
A.5. C OMPARISON AGAINST A DDITIONAL B ASELINES
We performed experiments over the same datasets using three new baselines:
ARMA (Bianchi et al.
BernNet (He et al.
and AdaGNN (Dong et al. (2021)).
Details about these baselines are given in Section A.5.2.
The results are reported in Tables 6 and 7.
The results are obtained after thorough optimization over the respective hyperparameters, the details of which are listed in A.5.2:
Test Acc
[Table] [Table]
For AdaGNN and BernNet, we use the authors code and tune it over the hyperparameters as provided in the paper.
For ARMA, we use the ofﬁcial PyTorch Geometric implementation (see A.5.2).
As a sanity check, we also tested ARMA on the node classiﬁcation datasets described in the paper and were able to reproduce similar numbers.
A.5. C OMPARISON AGAINST GENERAL FIR FILTERS
Instead of using a polynomial ﬁlter, we can use a general FIR ﬁlter (GFIR) which is described by the following equation:
[Equation]
where S is the graph shift operator (which in our case is (cid:101) A ), X is the node feature matrix and H k ’s are learnable ﬁlter matrices.
One can see GCN, SGCN, GPR-GNN as special cases of this GFIR ﬁlter, which constrain the H k in different ways.
We ﬁrst demonstrate that constraint on the GFIR ﬁlter is necessary for getting improvement in performance, particularly on heterophilic datasets.
Towards this, we build two versions of GFIR: one with regularization (constrained), and the other without regularization (unconstrained).
We ensure that that the number of trainable parameters in these models are comparable to those used in
PP-GNN.
We provide further details of the versions of the GFIR models below and report the results in Table 8 below:
• Unconstrained Setting :
In this setting, we do not impose any regularization constraints such as dropout and L2 regularization.
• Constrained Setting : In this setting, we impose dropouts as well as L2 regularization on the GFIR model.
Both dropouts and L2 regularization were applied on the H k ’s (the learnable ﬁlter matrices from the above equation).
We also compare PP-GNN (the proposed model) as well as GPR-GNN to the General FIR ﬁlter model (GFIR)
[Table]
We can make the following observation from the results reported in Table 8:
• Firstly, constrained GFIR performs better than the unconstrained version, with performance lifts of up to ∼ 10%.
This suggests that regularization is important for GFIR models.
• GPR-GNN outperforms the constrained GFIR version.
It is to be noted that GPR-GNN further restricts the space of graphs explored as compared to GFIR.
This suggests that regularization beyond simple L2/dropout kind of regularization (polynomial ﬁlter) is beneﬁcial.
• PP-GNN performs better than GPR-GNN.
Our model slightly expands the space of graphs explored (as compared to GPR-GNN, but lesser than GFIR), while retaining good performance.
This suggests that there is still room for improvement on how regularization is done.
PP-GNN has shown one possible way to constrain the space of graphs while improving performance on several datasets, however, it remains to be seen whether there are alternative methods that can do even better.
We hope to study and analyze this aspect in the future.
A.5. I MPLEMENTATION D ETAILS
In this subsection, we present several important points that are useful for practical implementation of our proposed method and other experiments related details.
Our approach is based on adaptation of eigen graphs constructed using eigen components.
Following Kipf & Welling (2017), we use a symmetric normalized version ( ˜ A ) of adjacency matrix A with self-loops: ˜
A =
˜ D − 12 ( A + I )
˜ D −where ˜ D ii = 1 + D ii , D ii
=
(cid:80) j A ij and ˜ D ij = 0 , i (cid:54) =
j .
We work with eigen matrix and eigen values of ˜ A .
To reduce the learnable hyper-parameters, we separately partition the low-end and high-end eigen values into several contiguous bins and use shared ﬁlter parameters for each of these bins.
The number of bins, which can be interpreted as number of ﬁlters, is swept in the range [2 , 3 , 4 , 5 , 10 , 20] .
The orders of the polynomial ﬁlters are swept in the range [1 , 10] in steps of 1 .
The number of EVD components are swept in the range [32 , 64 , 128 , 256 , 512 , 1024] .
In our experiments, we set η l = η h and we vary the η l parameter in range (0 , 1) and η gpr = 1 − η l .
For optimization, we use the Adam optimizer (Kingma & Ba, 2015).
We set early stopping to 200 and the maximum number of epochs to 1000.
We utilize learning rate with decay, with decay factor set to 0.99 and decay frequency set to 50.
All our experiments were performed on a machine with Intel Xeon 2.60GHz processor, 112GB Ram, Nvidia Tesla P-100 GPU with 16GB of memory,
Python 3.6, and PyTorch 1.9.0 (Paszke et al., 2019).
We used Optuna (Akiba et al., 2019) to optimize the hyperparameter search.
A.5. A DAPTABLE F REQUENCY R ESPONSES
In Figure 2 of the main paper, we observe that PP-GNN learns a complicated frequency response for a heterophilic dataset (Squirrel) and a simpler frequency response for a homophilic dataset (Citeseer).
We observe that this trend follows for two other datasets Chameleon (heterophilic) and Computer (homophilic).
See Figure 9.
[Figure]
A.5. D OES THE P OLYNOMIAL I NITIALIZATION MATTER FOR GIVING GOOD PERFORMANCE ?
In our experiments, we have varied the polynomial initialization schemes namely: [PPR, NPPR and Random], as described in GPRGNN paper Chien et al.
To understand whether this polynomial initialization is important, we perform an ablative study to compare the various initialization schemes.
We observe that our model (PP-GNN) retains similar performance across all initialization schemes.
For brevity, we report results on six datasets (Cora, Citeseer, Squirrel,
Chameleon, Texas and Wisconsin).
The results can be found in Table 9.
Our experiment results suggest that tuning polynomial initialization for each partition can indeed be avoided without compromising on the performance.
[Table]
A.5. D OES THE MLP EVEN MATTER ?
In PP-GNN there is a two layered MLP (that transforms the input node features) followed by a single graph ﬁltering layer similar to GPR-GNN. To understand MLP’s signiﬁcance, we ran an additional experiment, where we have used a single linear layer, instead of the two layered MLP. We continue to observe competitive (with respect to our original PP-GNN model) performance, across most datasets.
The results can be found in Table 10.
Also, the two layer MLP is not the signiﬁcant contributor towards performance.
This can also be seen by comparing GPR-GNN’s performance with that of LGC’s.
LGC can be interpreted as a linear version of GPR-GNN, and achieves comparable performance as GPR-GNN.
Note that PP-GNN (Linear) was not thoroughly trained as the PP-GNN (Original) due to time constraints.
The hyperparameter ranges are explained right below the next paragraph.
[Table]
Table 10 above seems to suggest that PP-GNN (Linear) is competitive to PP-GNN (Original).
We believe the difference in performance is an artifact of inadequate tuning.
We can infer that adding
MLP may give marginal improvements over the linear version.
This phenomenon is also observed in
GPR-GNN.
To illustrate this, we can compare GPR-GNN with LGC (linear version of GPR-GNN).
We can observe in the above table that GPR-GNN and LGC are comparable in performance.
For this additional set of experiments (Linear PP-GNN), due to time constraints, we have reduced the ranges for several hyper-parameters of our model, while also reducing the number of Optuna
Trials to 50.
These reduced ranges are as follows:
• Learning Rate: [0.003, 0.005] • Weight Decay : [0.0001, 0.001] • Dropout: [0.3, 0.5] • Hidden Dims: [32, 64] • Order of the Polynomial: [2, 4] • No. of Partitions (Buckets): [2, 4] • Number of Eigen Value: [256, 1024] • η : Sampled uniformly between [0, 1] using Optuna
Despite this smaller hyper-parameters search space, we are only shy by a few percentage points on the Texas dataset.
We believe this difference can be recovered with thorough tuning.
A. T RAINING T IME A NALYSIS
In the following subsections, we provide comprehensive timing analysis.
Computational Complexity: Listed below is the computational complexity for each piece in our model for a single forward pass.
Notation n : number of nodes, | E | : the number of edges, A : symmetric normalized adjacency matrix, F : features dimensions, d : hidden layer dimension, C : number of classes, e ∗ denotes the cost of EVD, K : polynomial order/hop order, l : number of eigenvalues/vectors in a single partition of spectrum (for implementation, we keep l same for all such intervals), m : number of partitions of a spectrum.
• MLP: O ( nFd + ndC ) •
GPR-term: O ( K | E | C ) + O ( nKC ) .
The ﬁrst term is the cost for computing A K f ( X ) for sparse A .
The second term is the cost of summation (cid:80) k
A k f ( X ) .
• Excess terms for PP-GNN: O ( mnlC ) .
This is obtained by the optimal matrix multiplication present in Equation 3 of the main paper ( U i is n × l , H i ( γ i ) is l × l , Z 0 () is n × C ).
The additional factor m is because we have m different contiguous intervals/different polynomials.
Note that typically n is much larger than l • EVD-term: e ∗ , the complexity for obtaining the eigenvalues/vectors of the adjacency matrix, which is usually very sparse for the observed graphs.
Most publicly available solvers for this task utilize Lanczos’ algorithm (which is a speciﬁc case of a more general Arnoldi iteration).
However, the convergence bound of this iterative procedure depends upon the starting vectors and the underlying spectrum (particularly the ratio of the absolute difference of two largest eigenvalues to the diameter of the spectrum)
[Saad (1980), LI (2010), Cullum & Willoughby (2002)].
Lanczos’ algorithm is shown to be a practically efﬁcient way for obtaining extreme eigenpairs for a similar and even very large systems.
We use ARPACK’s built-in implementation to precompute the eigenvalues/vectors for all datasets before training, thus amortizing this cost across training with different hyper-parameters conﬁguration.
Per Component Timing Breakup: In Table 11, we provide a breakdown of cost incurred in seconds for different components of our model.
Since the eigenpairs’ computation is a one time cost, we amortize this cost over the total hyper-parameters conﬁgurations and report the effective training time in the last column on of Table 11.
Average Training Time:
In Table 12, we report the training time averaged over 100 hyperparameter conﬁgurations for several models.
To understand the relative performance of our model with respect to GCN, we compute the relative time taken and report it in Table 13.
We can observe in Table 13 that PP-GNN is ∼ 4 x slower than GCN, ∼ 2 X slower than GPR-GNN and BernNET, and ∼ 2 X faster than AdaGNN.
However, it is important to note that in our average training time, the time taken to compute K top and bottom eigenvalues/vectors is amortized across the number of trials
Can we cut down the total training cost for PP-GNN?
To study this, we analyzed the chosen hyper-parameters for our 100 conﬁgurations experiments and prescribe below the new ranges for hyperparameters.
Reduced hyperparameters space:
• Learning Rate: [0.003, 0.005] • Weight Decay : [0.0001, 0.001] • Dropout: [0.3, 0.5] • Hidden Dims: [32, 64] • Order of the Polynomial: [2, 4] • No. of Partitions (Buckets): [2, 4] • Number of Eigen Value: [256, 1024] • η : Sampled uniformly between [0, 1] using Optuna
We further restrict the total number of Optuna trials to 20.
Note that as shown in Section A.5.7, our model is insensitive to the polynomial initialization scheme.
Hence we use ‘random’ initialization, thereby reducing the number of hyperparameters even further.
The total time taken by PP-GNN (including the one-time cost for obtaining eigenpairs) for training (and performing hyperparameter optimization) over 20 hyper-parameter conﬁgurations is reported in Table 14.
For comparison, we also show the total training time for the 100 conﬁgurations experiments in Table 15 which too includes the time taken to obtain min( n, 1024) eigenpairs.
From Tables [14,15] we can observe that the total training time has reduced by ∼ 5 x .
In Table 16, we can observe that even with the reduced hyper-parameter range, our model performs competitively or better.
A. D ISCREPANCIES IN P ERFORMANCE BETWEEN D IFFERENT GCN I MPLEMENTATIONS
We tested three different implementations of GCN(s)
(Kipf & Welling, 2017) provided by 1)
Thomas
Kipf and Max Welling 12 (In TensorFlow), 2) The implementation 13 of the authors of the GPR-GNN paper (Chien et al., 2021) (in PyTorch), 3) The code base 14 publicly available at the OGBN-ArXiv leader-board 15 (in PyTorch), on the OGBN-ArXiv dataset. [Table]
Since EVD is a one time cost
[Table]
we average this add
We
[Table]
Wisconsin 9.
Chameleon 14.
Squirrel 18.
Cora 12.
Citeseer [Table]
Cora-Full 24.
Pubmed 55.
[Table] Computers [Table] [Table]
Photo 8.
[Table] Flickr
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
WikiCS 14.
[Table] [Table]
[Table]
[Table] [Table] [Table]
Note that the authors of the leader-board code report a test accuracy of 71 .
74% on the OGBN-ArXiV dataset.
Extensive tuning was done on the Kipf and Welling implementation in order to match the results of the leaderboard code.
However, even after extensive hyper-parameter tuning, the test accuracy increased to only 65 .
53% , which is still much lesser.
Two other implementations that we tested were: a) The GCN implementation from the authors of the GPR-GNN paper (Chien et al., 2021) (in PyTorch) b)
The leader-board implementation (in
PyTorch).
The results for all the three implementations on the OGBN-ArXiv Dataset can be found in Table 17. [Table] [Table]
Besides different implementations of GCN(s), we ﬁnd that batch normalization is done in the implementation used in the GCN leader-board code.
We see that (in Table 17) the performance numbers vary depending on the implementation.
Also, batch normalization helps to get nearly 2% improvement for this dataset.
Since we did not perform batch normalization in all our experiments we ﬁnd a performance gap of ∼ 2% for this dataset in results reported in our paper (against the leader-board code).
We have also compared the GCN implementation from Kipf and Welling as well as the GCN implementation from the authors of the GPR-GNN across various datasets in Table 18.
We can see that both the TensorFlow and the PyTorch implementations lead to a similar performance for most of the datasets.
However, there are some differences in performance for some datasets (Wisconsin, Cora-Full (PCA), OGBN-ArXiv, Computer and Photos).
We have ensured that hyperparameters, preprocessing steps etc., are same across both these implementations.
We believe that these differences can be attributed to the different internal workings of TensorFlow and PyTorch.
A further study is required to understand where these gaps are coming from.
Note:
In the main paper (in Tables [1, 2]), we report the results obtained on the GCN implementation by the authors of the GPR-GNN paper. [Table] [Table]
[Table]
and Welling
[Table]
GCN (Kipf and Welling [Table] (Leaderboard Code) PyTorch
[Table]
GCN (Leaderboard Code)
PyTorch
[Table]
GCN (GPR-GNN)
PyTorch
[Table]
GCN (GPR-GNN)
PyTorch
False 69.
[Table]