G RAPH C ONVOLUTIONAL N ETWORKS VIA A DAPTIVE F ILTER B
ANKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph convolutional networks have been a powerful tool in representation learning of networked data.
However, most architectures of message passing graph convolutional networks (MPGCNs) are limited as they employ a single message passing strategy and typically focus on low-frequency information, especially when graph features or signals are heterogeneous in different dimensions.
Then, existing spectral graph convolutional operators lack a proper sharing scheme between ﬁlters, which may result in overﬁtting problems with numerous parameters.
In this paper, we present a novel graph convolution operator, termed BankGCN, which extends the capabilities of most MPGCNs beyond single ‘low-pass’ features and simpliﬁes spectral methods with a carefully designed sharing scheme between ﬁlters.
BankGCN decomposes multi-channel signals on arbitrary graphs into subspaces and shares adaptive ﬁlters to represent information in each subspace.
The ﬁlters of all subspaces differ in frequency response and together form a ﬁlter bank.
The ﬁlter bank and the signal decomposition permit to adaptively capture diverse spectral characteristics of graph data for target applications with a compact architecture.
We ﬁnally show through extensive experiments that BankGCN achieves excellent performance on a collection of benchmark graph datasets.
I NTRODUCTION
In many application domains, structured data is supported by graphs or networks.
To deal with such data, a collection of graph convolutional networks (GCNs), including MPGCNs and spectral
GCNs, have been proposed by generalizing architectures used for data in the Euclidean domain, to the irregular graphs.
Unfortunately, both types of architectures have implicit limitations when it comes to efﬁciently representing graph data information in terms of frequency ﬁltering.
MPGCNs (Gilmer et al., 2017; Velikovi et al., 2018; Hamilton et al., 2017; Xu et al., 2019) built on diverse message passing (MP) schemes are much prevalent with their ﬂexible and intuitive formulations of graph convolution.
The message passing schemes in classical MPGCNs, however, mostly focus on the low frequency characteristics of the data.
For example, GCN (Kipf & Welling, 2017) performs as Laplacian smoothing (Li et al., 2018), and the mean aggregator adopted in GraphSage (Hamilton et al., 2017) has naturally low-pass properties.
However, in addition to low-frequency components, graph data may also carry rich information in the middle- and high-frequency ranges.
There have been some recent attempts to complement low-pass features, either with predeﬁned ﬁlters (Gao et al., 2019;
Chang et al., 2021) or with other aggregation schemes (Abu-El-Haija et al., 2019; Zhu et al., 2020).
Yet, they still have some limitations, such as the nonadaptiviy of predeﬁned ﬁlters.
The importance of different frequency components may vary with the target tasks, it is thus beneﬁcial to be able to adapt the representation accordingly.
Spectral graph convolutional networks (Defferrard et al., 2016; Levie et al., 2018; Bianchi et al., 2021) are designed in the graph frequency domain directly and are much more powerful alternatives to process diverse spectral characteristics of graphs.
The lack of proper sharing schemes between ﬁlters, however, renders these models redundant and even prone to overﬁtting.
We go beyond the above limitations and propose to build effective adaptive representations for graph data with diverse spectral properties.
We design a novel graph convolution operator termed
BankGCN that utilizes an adaptive ﬁlter bank to process graph data in the frequency domain, as presented in Fig.
BankGCN provides an effective implementation to simplify spectral methods with a proper sharing scheme between ﬁlters with the help of signal decomposition.
Firstly, we decompose multi-channel graph signals into a collection of subspaces through projection, in order to adaptively separate input data according to signal characteristics.
Components in a subspace will then share a learnable ﬁlter that captures their particular frequency properties.
Notably, representations are built on ﬁnite impulse response (FIR) ﬁlters with a universal design (Tremblay et al., 2018) that correspond to local message passing schemes in the spatial domain.
Filters of all the subspaces together form a ﬁlter bank, and they are simultaneously learned from data together with subspace decomposition.
The ﬁlter bank representation is regularized to favour diversity in the frequency responses, in order to properly capture various spectral components in the graph signals.
The proposed convolution operator is stackable, and can be optimized together with other modules in the GCNs like graph pooling.
We validate the proposed architecture through extensive experiments on various graph classiﬁcation tasks.
BankGCN is more powerful than most MPGCNs, such as GCN (Kipf & Welling, 2017),
GraphSage
(Hamilton et al., 2017), and GIN (Xu et al., 2019), in that it is able to exploit more diverse spectral characteristics than ‘low-pass’ features in the data and adapts to its heterogeneous properties with a group of learnable multi-hop message passing strategies.
Furthermore, it outperforms its counterpart ChebNets (Defferrard et al., 2016) with a more compact architecture and achieves better generalization, and is superior to the most recent spectral method ARMA
(Bianchi et al., 2021).
It would be interesting to extend BankGCN to tasks like link prediction and applications into non-Euclidean data like 3-D point clouds.
R ELATED W ORK
We brieﬂy overview the main architectures for graph representation learning.
We describe ﬁrst existing message passing schemes, and then discuss several spectral methods.
Message Passing Graph Convolutional Networks.
MPGCNs design graph convolution with a variety of message passing schemes in the spatial domain.
For instance, messages are aggregated with the node-wise mean or max operation in a localized neighborhood in GraphSage (Hamilton et al., 2017), or based on attention scores in GAT (Velikovi et al., 2018).
A more expressive scheme, GIN, is further proposed with summing features in a neighborhood followed by a multi-layer perceptrons (MLP) to approximate any injective function on multiset (Xu et al., 2019).
However, these methods are typically constrained to a single one-hop message passing strategy that mostly captures ‘lowpass’ characteristics in the graph data.
Klicpera et al.
(2019) expand the MP range to multi-hop neighborhoods with graph diffusion operators. Besides, several methods are proposed to implement an equivalent high-order (polynomial) ﬁlter with the whole network to alleviate over-smoothing representation, including SGC (Wu et al., 2019), LGC (Navarin et al., 2020), GPR-GNN (Chien et al., 2021), SGF (NT et al., 2021), and S 2 GC (Zhu & Koniusz, 2021).
However, a single equivalent ﬁlter may be limited when representing multi-channel signals with diverse frequency characteristics. Alternatively, a group of works attempt to complement the ‘low-pass’ features using different strategies, such as graph wavelet ﬁlters (Gao et al., 2019; Chang et al., 2021), a one-order high-pass ﬁlter (Bo et al., 2021) and an additional quadratic middle-pass ﬁlter (Wu et al., 2021), and other aggregation schemes (Abu-El-Haija et al., 2019; Zhu et al., 2020).
Contrary to these methods, we employ a learnable ﬁlter bank with various frequency responses in a single layer to adaptively capture diverse spectral characteristics of signals rather than merely adding ‘high-pass’ features or speciﬁc components via predeﬁned graph wavelets.
Finally, in contrast with some works that further consider adaptive ﬁlters (Li et al., 2021; Dong et al., 2021;
Pasa et al., 2021), BankGCN is a more powerful and compact design with polynomial ﬁlter banks to avoid eigen-decomposition and a further sharing scheme between ﬁlters to reduce free parameters.
More detailed comparisons in App.
F. Filtering on Graphs and Spectral GCNs.
Filtering and Graph Fourier Transform have been generalized to graph data via spectral graph theory (Shuman et al., 2013).
Correspondingly, several spectral GCNs are derived from ﬁltering in the graph frequency domain directly, as pioneered by Bruna et al.
However, they rely on computationally expensive eigen-decomposition of the graph Laplacian. To avoid the eigendecomposition, similarly to dictionary learning methods (Thanou et al., 2014) in graph signal processing (GSP), the ﬁlters are built as polynomial or rational functions of eigenvalues of the graph Laplacian, such as Chebyshev polynomials in ChebNets (Defferrard et al., 2016) and TIGraNet (Khasanova & Frossard, 2017), Cayley polynomials in CayleyNets (Levie et al., 2018), and auto-regressive moving average (ARMA) ﬁlters (Bianchi et al., 2021).
In contrast with these spectral convolution methods that focus on the implementation of ﬁlters with desirable prop-
C on ca t e n a t e
ReLU s s s...
GFT
IGFT +    s...
GFT
IGFT +
GFT
IGFT +  (a) Framework.
(b) Filter bank.
(c) Message passing.
erties, like localization and narrowband specialization, this paper focuses on the design of the ﬁlter bank in order to construct a simpliﬁed architecture with improved generalization.
Finally, there are several papers in graph signal processing literature about the design of graph ﬁlter banks for graph signal decomposition (Narang & Ortega, 2012; Tanaka & Sakiyama, 2014) and multiscale analysis (Hammond et al., 2011; Narang & Ortega, 2013).
However, they usually work under a rigid constraint, e.g., perfect reconstruction, and thereby their produced (sparse) representations may not be ﬂexible enough to adapt to diverse tasks in machine learning.
In contrast, our method relaxes the perfect reconstruction condition and rather regularizes ﬁlters in a ﬁlter bank to be sufﬁciently different in the graph frequency domain.
P RELIMINARIES AND P ROBLEMS
In this paper, we consider data that is represented on undirected graphs.
Each vertex of graphs is attributed a d -channel signal or feature.
The node signal is typically multi-channel, i .
e ., d > 1 , and with different spectral statistics along different dimensions, which we refer to heterogeneous signals in this paper.
Notations and preliminaries on Graph Fourier Transform and ﬁltering are introduced in App. A.
We ﬁrst show how ﬁltering can be implemented by messaging passing, which is used in many stateof-the-art graph representation learning methods.
Filtering in the graph frequency domain and MP in the spatial domain is closely related (Shuman et al., 2013).
Taking the most popular GCN (Kipf &
Welling, 2017) as an example, according to its initial formation, the j -th channel of a ﬁltered signal is
[Equation]
where Θ denotes learnable parameters, d indicates the dimension of signals, and ˆ g (Λ) =
I − Λ is a low-pass ﬁlter (NT & Maehara, 2019;
Min et al., 2020).
The renormalized version of GCN and the
GIN can be formulated similarly.
Please refer to (Balcilar et al., 2021) for their speciﬁc equivalent supports of ˆ g (Λ) .
Besides, GraphSage uses the mean or max operator to aggregate information in neighborhoods.
Although it is hard to explicitly formulate its corresponding frequency response, the element-wise mean operation is a low-pass operation by nature and the element-wise max operator performs as an envelope extraction that suppresses the high-frequency components.
Another line of works, spectral GCNs, are directly derived from spectral ﬁltering.
For the representative ChebNets and CayleyNets, the ﬁlters are deﬁned as ˆ g ( λ ) =
(cid:80) Kk [-=
-]=0 θ
[Equation]
k ) T k ( λ ) in the graph frequency domain with the Chebyshev polynomial or Cayley polynomial basis { T k ( · ) } and corre- spondingly h [Equation] = d (cid:88)
[Equation]
U [Equation] where K is the order of the polynomial ﬁlters and { Θ ( k ) } are learnable parameters.
Notably, the above MPGCNs employ just a single kind of ﬁlter to handle all the channels of signals, and then takes different (linear) combinations of the ﬁltered signals to obtain different channels of the output signals.
Thereby, most MPGCNs are restricted in the number and types of ﬁlters and have a limited capacity in frequency-domain ﬁltering.
On the contrary, ChebNets and CayleyNets employ a different ﬁlter for each mapping from an input channel to an output channel and adopt a total of d × d (cid:48) K -order polynomial ﬁlters ( d and d (cid:48) denote the respective number of channels of input and output signals) , as presented in Eq.
This naive design neglects the potential relationship between ﬁlters and fails to reduce redundancy.
Consequently, they explore such a large number of learnable ﬁlters to process diverse frequency components of graph data at the expense of numerous free parameters, which sometimes leads to overﬁtting.
T HE B ANK GCN A LGORITHM
In this section, we ﬁrst outline the framework of BankGCN, then present its main elements, and ﬁnally discuss connections with spectral methods.
4. G RAPH C ONVOLUTION WITH A DAPTIVE F ILTER B ANKS
Signals (features) supported on nodes of a graph are usually high-dimensional and composed of multiple spectral patterns.
In other words, different signal channels vary differently over nodes, leading to diverse spectral characteristics.
To handle the heterogeneous signals, we employ a ﬁlter bank composed of a set of ﬁlters with different frequency responses in the design of BankGCN.
In order to further facilitate information processing, signal decomposition is adopted to explore latent relationships in the data.
For a multi-channel input node signal x ( v m ) , we ﬁrst adaptively decompose it into different subspaces and then employ different ﬁlters to deal with the signal components in each subspace separately.
The decomposition aims to map the components of signals with similar characteristics into the same subspace in order to facilitate the subsequent adaptive ﬁltering.
This is implemented with learnable subspace projections.
Mathematically, the input signal x ( v m ) is projected into s subspaces with a group of projection functions denoted by { f [ p ] ( · ) } ,
[Equation]
Here, the subscript “ [ p ] ” indicates the terms belonging to the p -th subspace and r [ p ] ( v m ) is the projected signal.
The choices for projection functions will be introduced in Section 4.2.
Subsequently, an adaptive ﬁlter g [ p ] ( · ) is designed within each subspace to represent the spectral characteristics of the corresponding signal components,
[Equation]
where we reuse ‘*’ to denote the convolution between each channel of signal r [ p ] and the ﬁlter g [ p ] .
The ﬁltered signals h
[ p ] ( v m ) , p = 1 , 2 , · · · , s , of all the subspaces are concatenated to produce the output features.
A Rectiﬁed Linear Unit (ReLU) is then used as a non-linear activation function,
[Equation]
) where the subscript l indicates variables or parameters in the l -th layer to describe the forward propagation between different layers in a hierarchical architecture.
Specially, considering that some ﬁlters are only supported on medium-to-high frequency components that usually mean large signal variations in the spatial domain, we further introduce a shortcut in each subspace that corresponds to full-pass in the graph frequency domain in order to make the ﬁltered signals stable.
Correspondingly, the ﬁltered signal h [ p ] ( v m ) in each subspace becomes
[Equation]
The shortcut can further facilitate back propagation of gradients, as for CNNs (He et al., 2016).
The proposed graph convolution operator is stackable, and the subspace mapping function in the following layers will further combine features from different subspaces in the preceding layers to enable information interaction between different channels.
4. S UBSPACE P ROJECTION
For the projection function f [ p ] ( · ) , there is a variety of design choices.
Here, we take the linear mapping as an example, since it is simple yet able to decompose different channels of signals.
Speciﬁcally, the d -channel 1 signal x ( v m ) is projected into s different subspaces with learnable matrices { W [ p ] } sp =1 .
For the sake of simplicity, all the subspaces have the same dimension in this paper.
To increase ﬂexibility, we further introduce a learnable bias b [ p ] for each subspace.
[Equation]
The introduction of subspace projection brings in two advantages: (i) it simpliﬁes the learning process.
Since the multi-channel graph signals have been decomposed, the ﬁlter in each subspace just needs to learn to capture spectral characteristics of the corresponding signal components; (ii) with low-dimensional subspaces, it limits the dimension of features output by the ﬁlter bank, and thereby reduces the number of free parameters and computation in the following layers.
4. F ILTERS
In order to handle graphs with arbitrary topologies and diverse signals, we adopt universal and adaptive ﬁlters to construct the ﬁlter bank.
For any function t : D = [0 , 2] → R , we can obtain a corresponding ﬁlter whose frequency response is ˆ g [ p ] ( λ ) =
t ( · ) , and its spatial construction is computed through IGFT (deﬁned as Eq. S2 in App. A):
[Equation]
with u i , i = 1 , · · · , n , the eigenvectors of the symmetric normalized graph Laplacian of a graph.
Notably, we directly design the frequency response of the ﬁlter for the continuous range
D = [0 , 2] in which the spectrum of an arbitrary graph locates as introduced in Section A.
In other words, given the discrete spectrum { λ 0 , λ 1 , . . . , λ n } of an arbitrary graph, we have λ i ∈ D and obtain the corresponding ﬁlter value on these speciﬁc discrete values ˆ g [ p ] ( λ i )
= t ( λ i ) , for i = 1 , 2 , . . . , n .
Thereby, the ﬁlter g [ p ] ( · ) is adaptable to any graph even with different topologies, i .
e ., it is a universal form (Tremblay et al., 2018; Levie et al., 2019).
Furthermore, we constrain the ﬁlter to the K -order polynomial function space in order to avoid the computation-intense eigendecomposition of the graph Laplacian, similarly to parametric dictionary learning (Thanou et al., 2014) in GSP.
It corresponds to an FIR ﬁlter.
Mathematically, the frequency response of a ﬁlter can be represented as
[Equation]
where { T k } denotes a speciﬁc polynomial basis such as Chebyshev polynomials, and { α ( k ) [ p ] } indicates the corresponding coefﬁcients.
With { α ( k ) [ p ] } learnable, we obtain an adaptive ﬁlter whose frequency response adapts to the data and to the target task.
Correspondingly, for the signal projected to the p -th subspace r [ p ] ( v m ) and R [ p ] =
[ r [ p ] ( v 1 ) , r [ p ] ( v 2 ) , . . .
, r [ p ] ( v n )] T , the ﬁltered signal is calculated as
[Equation] [Equation]
Equivalently, according to the relationship between frequency ﬁltering and localized linear transforms in the spatial domain (Shuman et al., 2013), the ﬁltering strategy corresponds to a message passing scheme within a K -hop neighborhood
[Equation]
with
[Equation]
Specially, the induced K -hop message passing scheme is learned from the data and exploits the multi-hop topological information of graphs through polynomials of the graph Laplacian.
Signal information is also taken into consideration through the learnable parameters { α ( k ) [ p ] } as well as in the subspace projection step.
More importantly, it permits to represent features that do not only have “low-pass” properties and explores the frequency components in a data-driven manner.
4. D IVERSITY R EGULARIZATION FOR F ILTER B ANKS
The ﬁlters constituting a ﬁlter bank should ideally have diverse frequency responses so that the signal is decomposed through ﬁltering into a series of signals with different frequency characteristics.
In
GSP, the ﬁlters are usually band-pass and divide the spectrum in different bands.
Considering that strict band-pass ﬁlters are difﬁcult to ﬁt through polynomial functions, we relax this strict bandpass requirement and rather target ﬁlters with diverse frequency responses, which we call “diversity condition”.
With the ﬁlter ˆ g
[ p ] ( λ ) given as a K -order polynomial function, the regularization on the ﬁlter is imposed on the respective polynomial coefﬁcients { α ( k ) [ p ] } p,k in Eq.
To achieve the diversity condition, we regularize the polynomial coefﬁcients to be well distributed in the parameter space.
Considering that the distances of the coefﬁcient vectors of two scaled ﬁlters may still be large in terms of the Euclidean distance, we thereby take the cosine distance to measure the distance between the polynomial coefﬁcients of ﬁlters.
Speciﬁcally, the regularization term is:
[Equation] [Equation]
where α [ p ] =
[ α (0)[ p ] , α (1)[ p ] , . .
, α ( K )
[ p ] ] T .
The max function reﬂects the maximum similarity between the polynomial coefﬁcients of any pair of ﬁlters in the ﬁlter bank.
Through minimizing Eq.
14, the most similar ﬁlters will have different orientations in the parameter space.
Thus, all the pairs of ﬁlters tend to be different.
When { T k ( L ) } k is an orthogonal basis such as Chebyshev polynomials, the diversity of polynomial coefﬁcients { α [ p ] } p implies that ﬁlters deﬁned as Eq.
10 are different in the graph frequency domain.
More intuitively, the message passing schemes in the spatial domain induced by the ﬁlters are different with diverse { α [ p ] } p , as presented in Eq.
12 and Eq. 13.
We can note that, if the ﬁlter is deﬁned on the basis composed of rectangular pulse functions 2 , i .
e .,
[Equation]
(15) the ideal subband ﬁlter banks, whose ﬁlters have different passbands in GSP, just corresponds to the optimal solution to the regularization with Ω( α ) = 0 in Eq.
14, when K ≥ s .
With T Θ ( G , Y ) generally representing a target function, the overall objective function is then formulated as
[Equation]
(16) where Y indicates ground truth labels, Θ denotes the parameter set including { α [ p ] , W [ p ] , b [ p ] } sp =1 , and γ is a hyperparameter to adjust the contribution of regularization term.
Like most popular graph convolution operators, BankGCN can be optimized via gradient-based methods together with modules such as graph pooling operators in GCNs.
It achieves linear computational complexity with
O ( K |E| d ) and constant learning complexity, similarly to most existing MPGCNs.
4. D ISCUSSION
BankGCN actually provides an effective simpliﬁcation of existing polynomial-based spectral methods with parameter decomposition.
With Θ =
[Θ 0 ; Θ 1 ; .
; Θ K ] ∈ R d × d (cid:48) ×
( K +1) representing the free parameters of the spectral methods adopting corresponding polynomial ﬁlters, BankGCN with a linear mapping as projection function actually provides a decomposition based approximation of Θ with
[Equation]
17) where ⊗ indicates outer product.
This decomposition strategy allows for ﬁlter sharing to reduce potential redundancy of existing spectral methods.
Multi-channel signals are ﬁrst adaptively decomposed into various subspaces with { W [ p ] } p , and signal components within a subspace share a ﬁlter deﬁned with α [ p ] to process corresponding characteristics.
In addition, it permits to further control the relationship between ﬁlters with the diversity regularization as introduced in Section 4.4.
We further study the capacity of BankGCN in handling the spectrum of signals with a compact architecture.
Proposition 1.
BankGCN can capture and preserve diverse frequency components of input graph signals, like its counterpart introduced in Eq.
Proof.
Please refer to App. B.
As demonstrated in Prop. 1 and its proof, a single ﬁlter g [ p ] in BankGCN together with signal projection is capable to explore and represent diverse frequency components of graph signals, as done in its counterpart with a group of ﬁlters { g ij } i .
This theoretically validates the effectiveness of BankGCN in reducing the number of ﬁlters and thereby potential redundancy in existing polynomial-based spectral architectures, while maintaining the capability of representing diverse spectral information.
E XPERIMENTS
In this section, in order to evaluate models in learning representation of diverse spectral information of graph data, we resort to node classiﬁcation and graph classiﬁcation tasks. Baselines.
We compare BankGCN with several state-of-the-art graph convolution methods, including MPGCNs and spectral methods, as listed in Table 1 and Table 2. Furthermore, two simpliﬁcations of the proposed model (BankGCN-Diff and BankGCN-NR) are considered for ablation study. Please refer to App. C & D for details.
[Table] We ﬁrst evaluate the proposed model in node classiﬁcation tasks on several heterophilic ( i [Table]
e ., neighbor nodes are dissimilar in labels/features) graph datasets, in order to validate the advantages of diverse frequency representation with ﬁlter banks. We follow the data split and evaluation procedure of (Pei et al., 2020; Zhu et al., 2020; Li et al., 2021), and describe the detail information of datasets and experimental settings in App. C. Results of baselines are cited from their respective publications if available, or recent public works (Zhu et al., 2020; Li et al., 2021). The metric β introduced by Pei et al.
(2020) reveals the degree of homophily of graph data in datasets (Small β corresponding to strong heterophily).
As presented in Table 1, BankGCN achieves the best performance on all the datasets.
In contrast, the MPGCNs that rely on low-pass features, like GCN and GAT, perform worse. Notably, BankGCN still outperforms several of the most recent MPGCNs that improve on the over-smoothing issues, including a simpliﬁed GCN (SGC), improved aggregation schemes MixHop, GEOM-GCN and H2GCN, and adaptive ﬁlter based methods SGF and ASGAT. [Table]
[Table] 5. G RAPH CLASSIFICATION
We further evaluate the different models in graph classiﬁcation tasks and study the components of
BankGCN.
Specially, for a fair and complete comparison, we consider two cases, (i) the same number of features, i .
e ., all of models with the same number of feature maps per hidden layer, and (ii) the same number of parameters, i .
e ., models with nearly the same number of free parameters per hidden layer, in Table 2 and Table 4 respectively. Experimental settings are presented in App.
D.
As presented in Table 2, BankGCN outperforms all the MP baselines with nearly the same or even fewer number of parameters on TU-benchmarks.
It further achieves better performance than its counterpart ChebNets with much fewer free parameters, i .
e ., about 1 / ( K + 1) , and is also superior to the most recent spectral method ARMA on most datasets.
As presented in Fig.
2, the learned ﬁlters have different frequency responses rather than only low-pass, some of them being high-pass and some focusing on middle-frequencies.
With such a bank of ﬁlters, BankGCN handles the multichannel signals ﬂexibly and thereby achieves the best performance on almost all the datasets.
We then go one step further to evaluate the adaptive ﬁltering capabilities.
As presented in Fig.
2, the learned ﬁlters have different frequency responses on various datasets as they are adapted to the data characteristics.
As listed in Tables 2 and 3, the BankGCN ( s = 1 ) employing one single adaptive ﬁlter still outperforms GCN with ‘low-pass’ ﬁltering on most datasets; Furthermore, BankGCN is superior to its variant BankGCN-Diff that uses predeﬁned ﬁlter banks on all the datasets.
These validate the beneﬁts of adaptive ﬁltering to ﬂexibly capture the spectral characteristics of data.
Study on the number of ﬁlters.
Furthermore, we evaluate the adoption of ﬁlter banks rather than a single ﬁlter, and study the impact of the number of ﬁlters in the ﬁlter bank (equivalently s , the number of subspaces) on the classiﬁcation performance.
With a group of ﬁlters, the ability of convolution operators to handle information is enhanced.
As presented in Table 2, BankGCN-Diff outperforms
GCN with additional band-pass and high-pass ﬁlters on almost all the datasets, and BankGCN further improves the performance with adaptive ﬁlters.
Furthermore, as s increases from 1 to 8 , the performance of BankGCN is improved on most datasets, as listed in Table 3.
These validate the beneﬁts of using more than one ﬁlter.
With s further increased into 16 , the performance is degraded on several datasets.
Given that the total dimension of all the subspaces is ﬁxed, the dimension of each space decreases and the representation capacity of each subspace probably declines with the further growth of s .
Study on the order of ﬁlters.
The order of polynomials determines the function space of ﬁlters.
In the graph frequency domain, as demonstrated in Fig.
2, it can better realize the bandpass property
λ 0 . 00 0 .0 .
50 1 .2 .|
ˆ g
[ p ] ( λ ) | (a) K = 2 ,γ = 0 .
λ 0 .0 .
5 2 .|
ˆ g
[ p ] ( λ ) | (b) K = 2 ,γ = 0 .
λ 0 .0 .
5 2 .|
ˆ g
[ p ] ( λ ) | (c) K
= 2 ,γ = 10 .
λ 0 .0 .
5 2 .|
ˆ g
[ p ] ( λ ) | (d) K = 3 ,γ = 10 .
λ 0 .0 .
5 2 .|
ˆ g
[ p ] ( λ ) | (e) K = 2 ,γ = 0 .
of ﬁlters using a larger K but at the cost of a greater risk of overﬁtting the spectrum of training data
In the spatial domain, the value of K corresponds to the message passing range, and a large
K will affect the locality of signals.
Thereby, a tradeoff is needed.
As shown in Table 3, BankGCN with K = 2 achieves the best performance on most datasets. We notice that for the cases with complex node signals, like FRANKENSTEIN with 780 -channel node attributes, a relatively large K is needed to exploit their various spectral characteristics; and the small K is preferred on simple datasets, such as the PROTEINS dataset with 3 -channel node category features. Study on the regularization.
We further evaluate the effect of the proposed diversity regularization.
The comparison of BankGCN-NR and BankGCN in Table 2 shows that the regularization improves the classiﬁcation performance on almost all the datasets.
On the FRANKENSTEIN dataset whose signal is composed of 780 -channel attributes, the regularization is not helpful.
We infer that the information in such high-channel signal is complex enough to induce different ﬁlters, as presented in Fig. 2(e).
Fig.
2(a)-(c) show that the learned ﬁlters in a ﬁlter bank with regularization present better diversity in frequency response, than those without regularization.
For example, the ﬁlters denoted by blue and red in Fig.
2(a) are with similar frequency responses, while they are more diverse in Fig. 2(b) and Fig. 2(c).
This is further veriﬁed by the maximum similarity scores of the polynomial coefﬁcients that deﬁne the ﬁlters Ω( α ) = 0 . 997 , 0 . 744 , and 0 .
649 (computed as Eq. 14) for Fig.
2(a)-(c), respectively.
More results and analysis are presented in App.
E.
[Table]
BankGCN still achieves the best performance on both CIFAR-10 and Ogbgmolhiv with all the models having a similar number of free parameters per hidden layer, as presented in Table 4.
Furthermore, to evaluate the generalization of models, we construct a reduced CIFAR-10 (1000) dataset by taking 100 graphs per category to form the training set, while maintaining the validation and testing sets.
BankGCN still performs best on the reduced CIFAR10 dataset and is among the models with the least performance loss compared with the full dataset.
Together with ROC-AUC being a measure of the generalization ability of a model, BankGCN performs well in the sense of generalization, especially when compared with its counterpart ChebNets.
These further validate the effect of the proposed sharing scheme between ﬁlters for model simpliﬁcation on improving generalization.
C ONCLUSION
In this paper, we propose a novel graph convolution operator, termed BankGCN, constructed on an adaptive ﬁlter bank for graph representation learning.
The ﬁlter bank is equivalent to a group of learnable message passing schemes in K -hop neighborhoods.
Together with subspace decomposition, BankGCN explores a sharing scheme between ﬁlters to adaptively handle information of diverse spectral characteristics with signiﬁcantly fewer parameters than its competitors, and achieves excellent performance on node classiﬁcation and graph classiﬁcation tasks.
An interesting direction for future research resides in discussing the capacity of the proposed graph convolution operator in terms of graph isomorphism test.
It may also be promising to employ BankGCN in a variety of tasks on non-Euclidean data like 3-D point cloud classiﬁcation and segmentation.
A N OTATIONS AND P RELIMINARIES
In this paper, we consider data that is represented on undirected graphs.
An input graph is described as G =
( V , E ) , with V and E respectively denoting the set of vertices and the set of edges.
We generally use capital letters for matrices and bold lowercase letters for vectors.
The graph topology is characterized by the adjacency matrix A with a non-zero value ( A ) ij indicating the weight of the edge connecting vertices v i and v j .
If available 3 , the d -channel signals or attributes of nodes are represented as x ( v m ) ∈ R d for ∀ v m ∈ V , and X =
[ x ( v 1 ) , x ( v 2 ) , · · · , x ( v n )] T is for the whole graph
G with n = |V| nodes.
Specially, for vector features, x i ( v m ) indicates the i -th channel of the signal on vertex v m , and x i =
[ x i ( v 1 ) , x i ( v 2 ) , · · · , x i ( v n )] T represents this signal on the whole graph.
For the hierarchical structure of GCNs, we employ a subscript l to indicate variables or parameters belonging to the l -th layer.
In most cases, we ignore l for clarity without causing confusion.
The Graph Fourier Transform (GFT) is deﬁned based on the graph Laplacian matrix L (Shuman et al., 2013).
We adopt the symmetric normalized graph Laplacian operator, i .
e ., L =
I − D − 12 AD − 12 with the diagonal degree matrix D deﬁned as ( D ) ii =
(cid:80) j ( A ) ij .
The eigendecomposion of the Laplacian matrix is denoted by L = U Λ U ∗ , where U =
[ u 1 , u 2 , . . .
, u n ] is composed of the eigenvectors u k , k = 1 , · · · , n and Λ is a diagonal matrix with eigenvalues
λ 1 , λ 2 , . .
. , λ n , with 0 = λ 1 ≤ λ 2 ≤ · · · ≤ λ n ≤ 2 .
The eigenvalues { λ k } construct the spectrum of the graph G 4 , and the GFT of a signal x is calculated as the inner-product between each of its component x i and the eigenvectors { u k } (Shuman et al., 2013):
[Equation]
where ∗ indicates conjugate transpose and u ∗ k ( v m ) denotes the m -th element in u ∗ k .
The Inverse
Graph Fourier Transform (IGFT) is deﬁned as
[Equation]
According to the convolution theorem, the convolution between the i -th channel of the signal x i ( v m ) and the corresponding ﬁlter g ( v m ) (with frequency response ˆ g ( λ k ) ) is deﬁned as:
[Equation]
Let us deﬁne ˆ g (Λ) =
diag([ˆ g ( λ 1 ) , ˆ g ( λ 2 ) , . . .
, ˆ g ( λ n )]) .
We have for x i ( v m ) that
[Equation]
Finally, a graph ﬁlter bank is composed of a set of ﬁlters { ˆ g i (Λ) } to decompose a graph signal into a series of signals with different frequency components (Narang & Ortega, 2012).
B P ROOF OF P ROPOSITION
Proof.
For spectral methods with polynomial ﬁlters ˆ g ( λ ) = (cid:80) Kk
=
θ ( k ) T k ( λ ) , the j -th channel ( j = 1 , 2 , . . .
, d (cid:48) ) of the output signal is h j = d (cid:88) i =
g
[Equation]
Thereby, the spectrum of output signal is a linear transform of the K -order polynomial expansion of input signals { T k (Λ)
[Equation]
i
[Equation]
For BankGCN, we assume that the j -th channel ( j = 1 , 2 , . . .
, d (cid:48) ) of the output signal corresponds to the q -th channel of the ﬁltered signal in the p -th subspace, and we have
[Equation]
where r [ p ] q denotes the q -th channel of the projected signal R [ p ] in the p -th subspace, b [ p ] q indicates the q -th element in b [ p ] , 1 denotes a constant vector with value 1, and (cid:15) [ p ] q represents a vector independent on input signals.
Correspondingly, the spectrum of h j is
[Equation]
According to Eq.
S6 and Eq. S8, BankGCN can capture and preserve diverse frequency components of input graph signals, like its counterpart introduced in Eq.
C E XPERIMENTAL S ETTINGS ON N ODE C LASSIFICATION
We evaluate the models on four heterophilic graph datasets (Pei et al., 2020; Tang et al., 2009), including Texas, Wisconsin, Cornell, and Actor.
Texas, Wisconsin, Cornell are three subdatasets from WebKB.
In these datasets, nodes represent web pages with their 1703-dimensional bag-ofwords representation as node features, and hyperlinks between the web pages form edges.
Actor is a dataset preprocessed by Pei et al.
(2020) from the ﬁlm-director-actor-writer network (Tang et al., 2009).
The graph in the dataset represents the co-occurrence (connection) of actors (nodes) on the same Wikipedia page. There are ﬁve categories on all these datasets. We follow the data split and evaluation procedure of previous methods (Pei et al., 2020; Zhu et al., 2020; Li et al., 2021). Speciﬁcally, 48%:32%:20% of nodes per class are split into training, validation and test sets, and 10 such random splits on each dataset are provided by Pei et al.
Fey & Lenssen (2019).
The network architecture consists of two layers.
We select the following hyper-parameters through grid-search: number of feature maps of the hidden layer ∈ { 32 , 64 , 128 } , learning rate ∈ { 0 . 02 , 0 . 04 } , weight decay ∈ { 5 e − 4 , 1 e − 3 } , and s ∈ { 4 , 8 , 16 } .
In addition, the dropout is 0 .
5 only in the hidden layer, and s is set as 1 for the output layer.
We compare BankGCN with several state-of-the-art graph convolution methods.
For the MPGCNs, we consider GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017), GAT (Velikovi et al., 2018), MixHop (Abu-El-Haija et al., 2019), SGC (Wu et al., 2019), GEOM-GCN (Pei et al., 2020),
H2GCN (Zhu et al., 2020), SGF (NT et al., 2021), and ASGAT (Li et al., 2021).
Regarding spectral GCNs, we compare with its counterpart ChebNets (Defferrard et al., 2016) that also adopts
Chebyshev polynomial ﬁlters, and the most recent spectral method ARMA (Bianchi et al., 2021).
D E XPERIMENTAL S ETTINGS ON G RAPH C LASSIFICATION
Datasets and data splits. For TU datasets (Kersting et al., 2016), we conduct experiments on seven widely used public benchmark graph classiﬁcation datasets, including ENZYMES, D&D, PROTEINS, NCI1, NCI109, MUTAGENICITY, and FRANKENSTEIN 5 .
We adopt node categorical features (one-hot encoding) and node attributes as node signals, depending on availability on the datasets, which are usually heterogeneous in different dimensions.
Speciﬁcally, node attributes are adopted on FRANKENSTEIN, node categorical features and node attributes (normalized in range [0 , 1] ) on ENZYMES, and node categorical features on the other datasets.
The statistics and properties of the datasets are summarized in Table S1.
According to (Lee et al., 2019), we use stratiﬁed sampling to randomly split each dataset into training, validation and test sets with a ratio of 8:1:1.
The trained model with the best validation performance is selected for test.
In order to alleviate the impact of data partition and network initialization, we conduct 20 random runs with different data splits and network initializations on each dataset, and report the mean accuracy with standard deviation of these 20 test results.
We further adopt two large benchmark datasets, CIFAR-10 (Dwivedi et al., 2020) and Ogbg-molhiv (Hu et al., 2020), in the experiments.
The graph version of CIFAR-10 is composed from the superpixels of images, and RGB intensities and normalized coordinates form node signals.
Ogbg-molhiv is a molecule graph dataset, with 9-dimensional node features including atomic number, chirality, and additional atom features.
We divide data on these two datasets in accordance with (Dwivedi et al., 2020) and (Hu et al., 2020), respectively.
Similarly, results are achieved with 3 runs on
CIFAR-10 and 10 runs on Ogbg-molhiv in order to alleviate the impact of network initialization.
Network architectures.
In the experiments, we adopt a similar architecture to (Xu et al., 2018), and the network consists of four convolution layers, one graph-level readout module and one prediction module for all the datasets.
Speciﬁcally, the graph convolution layer is designed as introduced in
Section 4 or deﬁned by various baseline models.
An l 2 normalization function is utilized in each convolution layer to stabilize and accelerate the training process, as in (Hamilton et al., 2017).
A graph-level readout module then aggregates the graph features from all the convolution layers to generate the graph representation h G .
It consists of node-wise mean and maximum operators (or mean operator on Ogbg-molhiv as in (Hu et al., 2020)), represented by ω ( · ) .
[Equation]
(S9)
[Table]
Finally, a prediction module composed of a linear fully connected layer and a softmax layer makes the prediction of the category of the input graph.
The cross-entropy (represented as T Θ ( G , Y ) ) is adopted as the loss function, which together with the regularization Ω( α ) for diversity condition composes the objective function:
[Equation]
where Θ denotes all the free parameters and Y indicates ground truth categorical labels.
In more details, we consider two versions of the architectures in the experiments in order to evaluate the models under the same number of features and the same number of parameters, respectively.
For the ﬁrst case, all of models have the same number of feature maps per hidden layer, whereas models are composed of nearly the same number of parameters per hidden layer for the latter case.
We consider the ﬁrst version with TU datasets in Table 2 with 64 feature maps per hidden layer, and the second version in Table 4 with nearly 16,500 and 65,800 free parameters per layer for all the models on CIFAR-10 and Ogbg-molhiv, respectively.
Conﬁgurations.
We implement the proposed models in Pytorch (Paszke et al., 2017) with geometric package (Fey & Lenssen, 2019), and optimize all of the models with the Adam optimizer (Kingma & Ba, 2015) on workstations with GPU GeForce GTX 1080 Ti for TU datasets as well as CIFARand with GPU RTX 2080 Ti for Ogbg-molhiv.
In more details, for TU datasets, the learning rate is 0.001 and the batch size is 64.
The number of training epochs is set as 500, and early stopping is employed with patience 30.
Finally, we obtain the following optimal hyper-parameters through grid search: weight decay ∈ { 0 , 1 e − 5 , 1 e − 4 } and γ ∈ { 0 , 0 .
For CIFAR-10 and Ogbg-molhiv, the batch size is increased to 256 due to their large scale.
Similarly to (Dwivedi et al., 2020), we adopt a dynamic learning rate that is initialized as 0.001 and decays by 0.1 when validation loss is not improved for 20 epochs until the minimum learning rate 1 e − 5 .
The number of training epochs is 500 with early stoping (patience 50).
The other settings are the same as TU datasets.
For the polynomial basis in the ﬁlter design of BankGCN, we adopt the widely used Chebyshev polynomials.
[Equation]
(S11)
The graph Laplacian is adopted as ˜ L
=
L − I for numerical stability, similarly to (Defferrard et al., 2016). Baselines.
We compare BankGCN with several state-of-the-art graph convolution methods. For the MPGCNs, we consider GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017) with mean aggregation, GAT ( 8 -heads) (Velikovi et al., 2018), and GIN using SUM-MLP (2 layers) that achieves the best performance (Xu et al., 2019). Regarding spectral GCNs, we compare with its counterpart ChebNets (Defferrard et al., 2016) that also adopts Chebyshev polynomial ﬁlters, and the most recent spectral method ARMA (1-stack) (Bianchi et al., 2021).
For a fair comparison, the results of baseline models are obtained with the same conﬁgurations as BankGCN using the public versions provided in the pytorch-geometric package (Fey & Lenssen, 2019), with their respective optimal hyper-parameters (weight decay) selected by grid search.
Ablation Study.
Furthermore, we consider two simpliﬁcations of the proposed model for ablation study.
First, BankGCN-Diff adopts a predeﬁned ﬁlter bank.
The ﬁlter bank consists of a lowpass ﬁlter, a high-pass ﬁlter, and band-pass ﬁlters from graph diffusion wavelets as used by Gama et al.
; Min et al.
The speciﬁc frequency responses of these ﬁlters are presented in
[Table] BankGCN-Diff adopts a predeﬁned ﬁlter bank that consists of a low-pass ﬁlter, a high-pass ﬁlter, and band-pass ﬁlters from graph diffusion wavelets as used in (Gama et al., 2019; Min et al., 2020). The speciﬁc frequency responses of these ﬁlters are
[Equation]
with two diffusion operators:
[Equation]
Their frequency responses can be easily derived with the relationship between the diffusion operators and the graph Laplacian.
Speciﬁcally, ψ l, 0 and ψ h, 0 are respective low-pass and high-pass ﬁlters, and the others are band-pass ﬁlters.
E M ORE E XPERIMENTAL R ESULTS AND ANALYSIS
More Ablation Results about the Diversity Regularization.
In Table S2, we consider the values of γ ∈ { 0 , 0 . 01 , 0 . 1 , 1 , 10 , 100 } to adjust its contribution in the objective function (Eq. 16).
The regularization improves the classiﬁcation performance on almost all the datasets.
Mostly, the best performance is achieved with γ = 0 .
1 or γ = 10 , and thereby we consider γ ∈ { 0 , 0 .
1 , 10 } in adjusting the contribution of regularization in the experiments.
As illustrated in Fig.
S1(a)-(f), the learned ﬁlters in a ﬁlter bank with regularization present better diversity in terms of frequency response than those without regularization on NCI109 and ENZYMES datasets.
For example, the ﬁlters denoted by blue and red in Fig.
S1(a) are with similar frequency responses, while they are more diverse in Fig.
S1(b) and Fig.
S1(c).
Besides, the ﬁlters indicated by gray and purple in
Fig.
S1(d)-(f) show similar results.
These are further veriﬁed with the maximum similarity scores of the polynomial coefﬁcients that deﬁne the ﬁlters Ω( α ) = 0 . 997 , 0 . 744 , and 0 .
649 (computed as
Eq.
(18)) for Fig.
S1(a)-(c), and Ω( α )
= 0 .
714 , and 0 .
705 for Fig.
S1(d)-(f), respectively.
More illustration results.
We provide more illustration results in Fig.
S2 to demonstrate adaptive ﬁlter banks with different numbers of ﬁlters.
With the increase of the s , BankGCN is more ﬂexible to decompose the signal into components with different spectral characteristics.
Furthermore, as shown in Fig.
S3, the frequency responses of the learned ﬁlters in a ﬁlter bank vary with the datasets and the layers in the neural networks.
These demonstrate the necessity and beneﬁt of adopting learnable rather than predeﬁned ﬁlters. F M ORE C OMPARISONS WITH R ELATED W ORKS
Message Passing Graph Convolutional Networks.
MPGCNs design graph convolution with a variety of message passing schemes in the spatial domain.
For instance, messages are aggregated with the node-wise mean or max operation in a localized neighborhood in GraphSage (Hamilton et al., 2017), or based on attention scores in GAT (Velikovi et al., 2018).
A more expressive scheme, GIN,
[Table]
[Figure] [Figure]
N
[Table] [Table]
M
[Table]
R
ANK
E N S
TE I N
[Table]
[Figure]
is further proposed with summing features in a neighborhood followed by a multi-layer perceptrons (MLP) to approximate any injective function on multiset (Xu et al., 2019).
However, these methods are typically constrained to a single one-hop message passing strategy that mostly captures ‘lowpass’ characteristics in the graph data.
Klicpera et al.
(2019) expand the MP range to multi-hop neighborhoods with graph diffusion operators.
Besides, several methods are proposed to implement a high-order (polynomial) ﬁlter with the whole network.
For example, SGC (Wu et al., 2019) corresponds to a ﬁxed polynomial ﬁlter, and LGC (Navarin et al., 2020), GPR-GNN (Chien et al., 2021), SGF (NT et al., 2021), and S 2 GC (Zhu & Koniusz, 2021) further realizes an adaptive ﬁlter with diverse motivations.
Alternatively, a group of methods attempt to complement the ‘low-pass’ features of GCN using different strategies.
For instance, Scattering GCN (Min et al., 2020) resorts to geometric scattering transform (Gao et al., 2019) to complement the ‘low-pass’ features of GCN (Kipf & Welling, 2017) with ‘band-pass’ features, a one-order high-pass ﬁlter is adopted as the complement (Bo et al., 2021), and SpGAT (Chang et al., 2021) employs predeﬁned graph wavelet ﬁlters.
Besides, MixHop (Abu-El-Haija et al., 2019) explores linear mixing of neighborhood information to further realize difference operators, which can capture high-frequency information, and H2GCN (Zhu et al., 2020)
additionaly aggregates information from high-order neighborhoods.
Contrary to these methods, we employ a learnable ﬁlter bank with various frequency responses to adaptively capture diverse spectral characteristics of signals rather than merely adding ‘high-pass’ features or speciﬁc components via predeﬁned graph wavelets.
Finally, some works further consider adaptive ﬁlters (Li et al., 2021;
Dong et al., 2021;
Pasa et al., 2021), but BankGCN is a more powerful and compact design with polynomial ﬁlter banks to avoid eigen-decomposition and a further consideration of the sharing scheme between ﬁlters to reduce free parameters.
For instance, AdaGNN (Dong et al., 2021) implements an equivalent ﬁlter for each channel of input signals through the whole network, while BankGCN can realize it within a single convolution operator.
Furthermore, the ﬁlter space of BankGCN with K -order polynomial ﬁlters is larger than AdaGNN, since the K -order ﬁlter in AdaGNN is obtained by multiplying K one-order ﬁlters.
The expressive capacity can be further enhanced by stacking multiple BankGCN layers.