S O GCN: S ECOND -O RDER G RAPH C ONVOLUTIONAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce a second-order graph convolution (SoGC), a maximally localized kernel, that can express a polynomial spectral ﬁlter with arbitrary coefﬁcients.
We contrast our SoGC with vanilla GCN, ﬁrst-order (one-hop) aggregation, and higher-order (multi-hop) aggregation by analyzing graph convolutional layers via generalized ﬁlter space.
We argue that SoGC is a simple design capable of forming the basic building block of graph convolution, playing the same role as 3 ×kernels in CNNs.
We build purely topological Second-Order Graph Convolutional
Networks (SoGCN) and demonstrate that SoGCN consistently achieves state-ofthe-art performance on the latest benchmark.
Moreover, we introduce the Gated
Recurrent Unit (GRU) to spectral GCNs.
This explorative attempt further improves our experimental results.
I NTRODUCTION
Deep localized convolutional ﬁlters have achieved great success in the ﬁeld of deep learning.
In image recognition, the effectiveness of 3 × 3 kernels as the basic building block in Convolutional
Neural Networks (CNNs) is shown both experimentally and theoretically (Zhou, 2020).
We are inspired to search for the maximally localized Graph Convolution (GC) kernel with full expressiveness power for Graph Convolutional Networks (GCNs).
Most existing GCN methods utilize localized GCs based on one-hop aggregation scheme as the basic building block.
Extensive works have shown performance limitations of such design due to over-smoothing (Li et al., 2018;
Oono & Suzuki, 2019; Cai & Wang, 2020).
In vanilla GCNs (Kipf & Welling, 2017) the root cause of its deﬁciency is the lumping of the graph node self-connection with pairwise neighboring connections.
Recent works of Xu et al.
; Dehmamy et al.
Ming Chen et al.
(2020) disentangle the effect of self-connection by adding an identity mapping (so-called ﬁrst-order GC).
However, its lack of expressive power in ﬁlter representation remains (Abu-El-Haija et al., 2019).
The work of (Ming Chen et al., 2020) conjectured that the ability to express a polynomial ﬁlter with arbitrary coefﬁcients is essential for preventing over-smoothing.
A longer propagation distance in the graph facilitates GCNs to retain its expressive power, as pointed out by (Liao et al., 2019;
Luan et al., 2019; Abu-El-Haija et al., 2019).
The minimum propagation distance needed to construct our building block of GCN remains the open question.
We show that the minimum propagation distance is two: a two-hop graph kernel with the second-order polynomials in adjacency matrices is sufﬁcient.
We call our graph kernel Second-Order GC (SoGC).
We introduce a Layer Spanning Space (LSS) framework to quantify the expressive power of multilayer GCs for modeling a polynomial ﬁlter with arbitrary coefﬁcients.
By relating low-pass ﬁltering on the graph spectrum (Hoang & Maehara, 2019) with over-smoothing, one can see the lack of ﬁlter representation power (Ming Chen et al., 2020) can lead to the performance limitation of GCN.
Using the LSS framework, we show that SoGCs can approximate any linear GCNs in channel-wise ﬁltering.
Furthermore, higher-order GCs do not contribute more expressiveness, and vanilla GCN or ﬁrst-order GCs cannot represent all polynomial ﬁlters in general.
In this sense, SoGC is the maximally localized graph kernel with the full representation power.
To validate our theory, we build Second-Order Graph Convolutional Networks (SoGCN) using
SoGC kernels.
Our SoGCN using simple graph topological features consistently achieves state-
[Figure]
To our best knowledge, this work is the ﬁrst study that identiﬁes the importance of the two-hop neighborhood in the context of GCNs’ ability to express a polynomial ﬁlter with arbitrary coefﬁcients.
Our model is a special but non-trivial case of Defferrard et al.
Kipf & Welling (2017) conducted an ablation study with GC kernels of different orders but missed the effectiveness of the second-order relationships.
The work of Abu-El-Haija et al.
(2019) talked about muti-hop graph kernels; however, they did not identify the critical importance of the second-order form.
In contrast, we clarify the prominence of SoGCs in theories and experiments.
Our research on graph convolution using pure topologically relationship is orthogonal to those uses geometric relations (Monti et al., 2017; Fey et al., 2018;
Pei et al., 2020), or those with expressive edge features (Li et al., 2016; Gilmer et al., 2017; Corso et al., 2020), and hyper-edges (Morris et al., 2019; Maron et al., 2018; 2019).
It is also independent with graph sampling procedures (Rong et al., 2019; Hamilton et al., 2017; Li et al., 2019).
P RELIMINARIES
We begin by reformulating spectral GCNs and introducing our notation.
We are interested in a ﬁnite graph set G = (cid:8) G 1 , · · · , G |G| (cid:9) .
Assume each graph G ∈ G is simple and undirected, associated with a ﬁnite vertex set V ( G ) , an edge set E ( G ) =
{ ( u, v ) : ∀ u ↔ v } , and a symmetric normalized adjacency matrix A ( G ) (Chung & Graham, 1997; Shi & Malik, 2000).
Without loss of generality and for simplicity, |V ( G ) | = N for every G ∈ G .
Single-channel features x ∈ R N supported in graph G ∈ G is a vectorization of function V ( G ) → R .
Graph Convolutions (GCs) is known as Linear Shift-Invariant (LSI) operators to adjacency matrices (Sandryhaila & Moura, 2013).
By this deﬁnition, GCs can extract features regardless of where local structures fall.
Given parameter space Ω ⊆ R , we write a single-channel GC (Sandryhaila & Moura, 2013; Defferrard et al., 2016) as a mapping f θ : G × R N → R N such that 1 :
[Equation]
where θ =
[ θ· · · θ K ] T ∈
Ω K +1 parameterizes the GC.
K reﬂects the localization of f θ : a linear combination of features aggregated by A ( G ) k .
Moreover, we reformulate two popular models, vanilla GC (Figure 1a) and ﬁrst-order GC (Figure 1b), as below:
[Equation]
The general spectral GCNs stack L layers of GCs (Equation 1) with nonlinear activations.
Let f ( l ) be GC layers with parameters θ
( l ) ∈
Ω K +1 , l ∈
[ L ] , the single-channel GCNs can be written as:
[Equation]
[Figure]
u
[Figure]
e
[Figure]
where σ is an element-wise activation function, the superscripts denote the corresponding layer number, g is a task-speciﬁed readout function (e.g., softmax), the inputs are graph G ∈ G and signals x ∈ R N .
The compositionality principle of deep learning suggests L being large, while K being small and localized (LeCun et al., 2015).
O VERVIEW : S ECOND -O RDER G RAPH C ONVOLUTION
We are interested in the overall graph convolution network’s representation power of expressing a polynomial ﬁlter (Equation 1) with arbitrary coefﬁcients.
A multi-layer GCN approximate a Korder polynomial ﬁlter (cid:80) Kk =
θ k A ( G ) k , θ k ∈ Ω , k = 0 , · · · K , by stacking basic building blocks of graph convolution (GC) layers (Wu et al., 2019; Ming Chen et al., 2020).
We formally deﬁne the second-order GC (SoGC) using the second-order polynomials of adjacency matrices:
[Equation]
(4) where θ i ∈ R , i = 0 , 1 , 2 are trainable paremeters in the context of machine learning.
Its vertexdomain interpretation is illustrated in Figure 1c.
At ﬁrst glance, it seems that we could stack two one-hop graph convolution (GC) kernels to approximate a SoGC.
However, as shown in Section 4.3, that is not the case.
The critical insight is that graph ﬁlter approximation can be viewed as a polynomial factorization problem.
It is known that any univariate polynomial can be factorized into sub-polynomials of degree two.
Based on this fact, we show by stacking enough SoGCs (and varying their parameters) can achieve decomposition of any polynomial ﬁlters.
In contrast, ﬁrst-order GCs are not universal approximators; two stacked one-hop GCs cannot model every two-hop GC.
Polynomial ﬁlter completeness of SoGC leads to better performance of GCNs.
As shown in Figure 2, networks built with SoGC can overcome over-smoothing and extract features on high-frequency bands.
In the next section, we demonstrate our formal arguments on polynomial approximation.
R EPRESENTATION P OWER A NALYSIS 4. L AYER S PANNING S PACE F RAMEWORK
To illustrate the representation power of GC layers, we establish a Layer Spanning Space (LSS) framework to study the graph ﬁlter space spanned by stacking multiple graph kernels.
First, we present our mathematical devices in Deﬁnition 1, 2 with Lemma 1 as below.
Deﬁnition 1.
Suppose the parameter space Ω = R .
The Linear Shift-Invariant (LSI) graph ﬁlter space of degree K > 0 with respect to a ﬁnite graph set G is deﬁned as A = (cid:8)
f θ : G × R N → R N , ∀ θ ∈ R K +1 (cid:9) , where f θ follows the deﬁnition in Equation 1.
Deﬁnition 2.
Let spectrum set S ( G )
= { λ : λ ∈ S ( A ( G )) , ∀ G ∈ G} , where S ( A ) denotes the eigenvalues of A .
Deﬁne spectrum capacity Γ =
|S ( G ) | .
In particular, Γ = ( N − 1) |G| if every graph adjacency matrix has no common eigenvalues other than 1 .
Lemma 1.
A of degree K > 0 has dimension min { K + 1 , Γ } as a vector space.
Proof of Lemma 1 follows from Theorem 3 of Sandryhaila & Moura (2013).
The complete version can be found in Appendix C. Here, we induce a ﬁnite-dimension ﬁlter space by Lemma 1.
For simplicity, we will model the linear composition of ﬁlters to analyze its representation power.
The nonlinear activation effects are beyond the scope of this work.
Following Deﬁnition 1, let A be the full ﬁlter space of degree Γ − 1 and B be the low-level ﬁlter space as a set of polynomials in adjacency matrices (Equation 1).
Denote the GC at l -th layer by f ( l ) ∈ B , then we yield the LSS of stacking L layers as below:
[Equation]
where p ( l ) ( x ) varies in a certain class of polynomials.
We can assess the expressive capability of
GC layers by comparing the LSS with A .
Kernels in B have full representation power if A ⊆ B L .
We are interested in B K , which denotes all localized ﬁlters of degree at most K .
The LSS of B K is modeled as:
[Equation]
where the number of layers is bounded by L ≤ (cid:100) (Γ − 1) /K (cid:101) , according to Lemma 1.
4. U NIVERSAL R EPRESENTATION P OWER OF S O GC
In this section, we present Theorem 1 to demonstrate the universal representation power of SoGCs as claimed in Section 3.
Formally, we add superscripts to Equation 4 to indicate the layer number.
Then we leverage a fundamental polynomial factorization theorem to conclude Theorem 1 as below.
Theorem 1.
For any f ∈ A , there exists f ( l ) 2 ∈ B 2 with coefﬁcients θ ( l ) 0 , θ ( l ) 1 , θ ( l ) 2 ∈ R , l = 1 , · · · , L such that f = f ( L )◦ · · · ◦ f (1) where L ≤ (cid:100)
(Γ − 1) / 2 (cid:101) .
The complete proof is presented Appendix D. Theorem 1 can be regarded as the universal approximation theorem of linear GCNs, which implies multi-layer SoGCs have full ﬁlter expressiveness.
Theorem 1 also demonstrates how GCNs with SoGCs beneﬁt from depth, which coincides with the view of Dehmamy et al.
Figure 3a veriﬁes our SoGCN can overcome over-smoothing and successfully utilize depth to attain performance gain.
4. R EPRESENTATION P OWER OF O THER G RAPH C ONVOLUTION
In this section, we show that vanilla and ﬁrst-order GCs lack expressiveness, while higher-order GCs reduce compactness and increase ﬁtting difﬁculty.
Vanilla vs. second-order.
Extensive works have shown the performance deﬁciency of vanilla
GCNs (Hoang & Maehara, 2019; Luan et al., 2019; Oono & Suzuki, 2019; Cai & Wang, 2020).
Based on the LSS framework, we can point out a similar issue but from a novel perspective.
Let us write f ( l )
( G, x )
= θ ( l ) ( A ( G ) + I ) x ∈ B 0 as the l -th GC layer 2 .
Then L of them can represent a
LSS as follows:
[Equation]
by letting θ = θ ( L ) · · · θ (1) .
No matter how large L is or how a optimizer tunes the parameters θ ( l ) , dim B L 0 = 1 which signiﬁes B L 0 degenerates to a negligible subspace of A .
First-order vs. second-order.
We denote ﬁrst-order GCs as f ( l ) 1 ( G, x ) =
( θ ( l ) 1 A ( G )+ θ ( l ) {+
+}
I ) x ∈ B 1 .
In the spirit of Section 4.1, write the LSS as:
[Equation]
which is isomorphic to a polynomial space whose elements split over the real domain.
Compared with B L 0 (Equation 7), B L 1 represents a much larger subset of A .
This highlights the importance of the ﬁrst-order term or the identity mapping mentioned in (Xu et al., 2019; Dehmamy et al., 2019;
Ming Chen et al., 2020).
The limitations also become obvious since not all polynomials can be factorized into ﬁrst-order polynomials.
These polynomials only occupy a small proportion in the ambient polynomial space (Li, 2011), which indicates ﬁrst-order GCs are not universal approximators in general.
Higher-order vs. second-order.
GCs of degree K ≥ 2 are called higher-order GCs. They can model multi-hop GCNs such as Luan et al. ; Liao et al.
; Abu-El-Haija et al.
Higher-order GCs have equivalent expressive power to SoGCs, since they can be reduced to
SoGCs as long as coefﬁcient sparsity can be achieved.
But this by-product–an uncertain sparsity of coefﬁcients–is not compatible with gradient-based optimization algorithms.
Extensive experiments (Defferrard et al., 2016) have shown the ineffectiveness of learning higher-order kernels, because eigenvalues of graph adjacency matrices diminish when powered.
This results in a decreasing numerical rank of A ( G ) k , which prevent higher-order GCs from aggregating larger-scale information.
SoGCs can alleviate this problem by preventing the loss of information due to higher-order powering operation.
Finally, higher-order GC lacks nonlinearity.
SoGCN can bring a better balance between the expressive power of low-level layers and nonlinearity among them.
S ECOND -O RDER G RAPH C ONVOLUTIONAL N ETWORKS
In this section, we introduce other building blocks of GCNs and establish our Second-Order Graph Convolutional Networks (SoGCN) following the fashion of deep learning. First, we promote SoGC to the multi-channel version analogous to Kipf & Welling (2017).
Then we cascade a feature embedding layer, multiple SoGC layers, and append a readout module.
Suppose the multi-channel input is X ∈ R N × D supported in graph G ∈ G , denote the output of l -th layer as X ( l ) ∈ R N × E , the ﬁnal node-level output as Y ∈ R N × F , or graph-level output as Y ∈ R E , we formulate our novel deep GCN based on SoGC as follows:
[Equation]
[Equation]
where Θ ( l ) i ∈
R E × E , i = 0 , 1 , 2 are trainable weights for linear ﬁlters; ρ : R N × D → R N × E is an equivariant embedder (Maron et al., 2018) with parameters Φ ; σ : R N × E
→ R N × E is an activation function.
For node-level readout, τ : R N × E → R N × F can be a decoder (with parameters Ψ ) or a nonlinear activation (e.g., softmax) in place of the prior layer.
For graph-level output, τ : R N × E → R E should be an invariant readout function (Maron et al., 2018), e.g., channel-wise sum, mean or max (Hamilton et al., 2017).
In practice, we adopt ReLU as nonlinear activation (i.e., σ = ReLU ), a multi-layer perceptron (MLP) as the embedding function ρ , another MLP for node regression readout, and sum (Xu et al., 2019) for graph classiﬁcation readout.
5. G ATED R ECURRENT U NIT
Gated Recurrent Unit (GRU) has been served as a basic building block in message-passing GNN architectures (Li et al., 2016; Gilmer et al., 2017; Corso et al., 2020).
In this subsection, we explore its application in spectral GCNs.
According to Cho et al.
(2014), GRU can utilize gate mechanism to preserve and forget information.
We hypothesize that a GRU can be trained to remove redundant signals and retain lost features on the spectrum.
This function can be used to alleviate the oversmoothing problem of vanilla GCNs by maintaining information from previous layer and canceling the dominance of low-frequencies.
By the same means, GRU can also relieve the side-effect of ReLU, which is proved to be a special low-pass ﬁlter (Oono & Suzuki, 2019; Cai & Wang, 2020).
Even though piled-up SoGCs attain full expressiveness, we show by our experiment that GRU can still facilitate SoGCN in avoiding noises and enhancing features on the spectrum (Figure 2)
Similar to Li et al.
; Gilmer et al.
(2017), we appends a shared GRU module after each GC layer, which takes the signal before the GC layer as the hidden state, after the GC layer as the current input.
We note that GRU can cooperate with any spectral GCs (Equation 1).
When integrated with SoGCN, we call this special variant SoGCN-GRU.
We formulate its implementation by replacing Equation 10 with Equation 12 as below.
[Equation]
(12) where X ( l +1) conv is the input, X ( l ) represents the hidden state, Ω denotes parameters of the GRU.
Figure 2 and Figure 5 verify our conjecture.
We observe more steady low-frequency component on the spectrum head and more characteristical bands on the high-frequency tail.
Our empirical study in Table 3 also indicates the effectiveness of GRU for spectral GCNs in general.
Hence, we suggest including this recurrent module as another basic building block of our SoGCNs.
5. C OMPARISON TO R ELATED W ORK
Spectral GCNs.
Spectral GCN leverages polynomials in the adjacency matrix to represent graph convolutional layers (Bruna et al., 2014).
Many works have been discussing how to design the polynomial and choose its degree to compose a localized GC layer.
ChebyNet (Defferrard et al., 2016) approximates graph ﬁlters using Chebyshev polynomials.
Vanilla GCN (Kipf & Welling, 2017; Wu et al., 2019) further reduces the GC layer to a degree-one polynomial with ﬁrst-order and constant terms merged.
However, these simpliﬁcations cause over-smoothing and performance loss.
Our SoGC incorporates only one hop longer but obtains the full representation power.
This design keeps each layer localized, simple and easy to implement, but makes the whole GCN much more powerful.
Our work reveals the critical degree of polynomial ﬁlters where kernel size is minimized while ﬁlter representation power is maximized.
Multi-Hop GCNs.
To exploit multi-scale information, Luan et al.
(2019) devises snowball GCN and truncated Krylov GCN to capture neighborhoods at different distances.
To simulate hop delta functions, Abu-El-Haija et al.
(2019) repeat mixing multi-hop features to identify more topological information.
These models exhibit the strength of multi-hop GCNs over one-hop GCNs while leaving the propagation length as a hyper-parameter.
Modeling those multi-hop GCNs as our higherorder models, SoGCN possesses the identical representation power but has ﬁxed size and better localization, making SoGC more suitable to be the basic building block in GCNs.
It is noteworthy that, although Abu-El-Haija et al.
(2019) investigates the two-hop delta function (a Gabor-like ﬁlter), their ﬁnal proposed solution is only a generic class of multi-hop GCNs.
The discussion on two-hop delta functions cannot attain our theoretical results.
Expressiveness of GCNs.
Most of works on GCN’s expressiveness are restricted to the oversmoothing problem: Li et al.
ﬁrst poses the over-smoothing problem; Hoang & Maehara (2019) indicates GCNs are no more than low-pass ﬁlters; Luan et al.
; Oono & Suzuki (2019) demonstrate the asymptotic behavior of feature activation to a subspace; Cai & Wang (2020) examines the decreasing Dirichlet energy.
Unlike them, our established LSS framework can identify speciﬁc issues of GCNs with algebraic and geometric interpretation.
The over-smoothing problem can be formulated as one of our sub-problems (Section 4.3).
In this sense, SoGCN solves a more general expressiveness issue than those relieving over-smoothing problem only (Xu et al., 2018;
Rong et al., 2019; Chen et al., 2020; Ming Chen et al., 2020).
We investigate the overall expressive power of GCNs by discussing ﬁlter completeness.
This direction is orthogonal to those studying message-passing GNNs (Scarselli et al., 2008) and Weisfeiler-Leman GNNs (Xu et al., 2019; Morris et al., 2019).
E XPERIMENTS
Experiments are conducted on the synthetic dataset in Section 6.1 and on the GNN benchmarks (Dwivedi et al., 2020) in Section 6.2.
6. S YNTHETIC G RAPH S PECTRUM D ATASET FOR N ODE R EGRESSION
To validate the expressiveness of SoGCN, and its power to preserve higher-order graph signal, we build a Synthetic Graph Spectrum (SGS) dataset for the node signal ﬁltering regression task.
We construct SGS dataset with random graphs.
The learning task is to mimic three types of hand-crafted ﬁltering functions: high-pass, low-pass, and band-pass on the graph spectral space (deﬁned over the graph eigenvectors).
There are 1k training graphs, 1k validation graphs, and 2k testing graphs for each ﬁltering function.
Each graph is undirected and comprises 80 ˜ 120 nodes.
Appendix E covers more details of our SGS dataset.
We choose Mean Absolute Error (MAE) as the evaluation metric.
Experimental Setup.
We compare SoGCN with vanilla GCN (Kipf & Welling, 2017), ﬁrst-order
GCN, and higher-order GCNs on the synthetic dataset.
To evaluate each model’s expressiveness purely on the graph kernel design, we remove ReLU activations for all tested models.
We adopt the Adam optimizer (Kingma & Ba, 2015) in our training process, with a batch size of 128.
The learning rate begins with 0.01 and decays by half once the validation loss stagnates for more than 10 training epochs.
[Table]
Results and Discussion.
Table 1 summarizes the quantitative comparisons.
SoGCN achieves the superior performance on all of the 3 tasks outperforming vanilla GCN and 1st-order GCN, which implies that SoGC graph convolutional kernel does beneﬁt from explicit disentangling of the θ 0 I (zero-hop) and θ 2 A 2 (second-hop) terms.
Our results also show that higher-order (third-order and fourth-order) GCNs do not improve the performance further, even though they have many more parameters.
SoGCN possesses a more expressive representation ability and a good trade-off between performance and model size.
Figure 3 plots MAE results as we vary the depth and channel width of GC layers.
Vanilla GCN can beneﬁt from neither depth nor width.
First-order GC, SoGC
[Table]
and higher-order GC can leverage depth to span larger LSS.
Figure 3a illustrates the corresponding performance for each graph kernel types.
SoGC and higher-order GC both outperform ﬁrst-order GC as depth increases.
Figure 3b shows the beneﬁts of SoGC remain as we move to multi-channel construction.
Comparing Figure 3a and 3b, we ﬁnd that depth has larger effect on GCNs. [Figure]
6. GNN B ENCHMARKS
We follow the benchmarks outlined in Dwivedi et al.
(2020) for evaluating GNNs on several datasets across a variety of artiﬁcial and real-world tasks.
We choose to evaluate our SoGCN on a real-world chemistry dataset (ZINC molecules) for the graph regression task, two semi-artiﬁcial computer vision datasets (CIFAR10 and MNIST superpixels) for the graph classiﬁcation task, and two artiﬁcial social network datasets (CLUSTER and PATTERN) for the node classiﬁcation task.
Experimental Setup.
We compare our proposed SoGCN and SoGCN-GRU with state-of-the-art
GNNs: vanilla GCN (Kipf & Welling, 2017), GAT (Veli ˇ ckovi ´
c et al., 2018)
, MoNet (Monti et al., 2017), GIN (Xu et al., 2019), GraphSage (Hamilton et al., 2017), GatedGCN (Bresson & Laurent (2017)) and 3WL-GNN (Maron et al., 2019).
To ensure fair comparisons, we follow the same training and evaluation pipelines (including optimizer settings) and data splits of benchmarks.
Furthermore, we adjusted our model depth and width to ensure it satisﬁes parameter budgets as speciﬁed in the benchmark.
Note that we do not use any geometrical information to encode rich graph edge relationship, as in models such as GatedGCN-E-PE.
We only employ graph connectivity information for all tested models.
[Table] [Table] Results and Discussion. Table 2 reports the benchmark results.
Our model SoGCN makes small computational changes to GCN by adopting second-hop and zero-hop neighborhoods, and it outperforms models with complex message-passing mechanisms.
With GRU module, SoGCN-GRU tops almost all state-of-the-art GNNs on the ZINC, MNIST and CIFAR10 datasets.
Whereas, GRU does not lift performance on the CLUSTER and PATTERN datasets for node classiﬁcation task.
As suggested by Li et al.
(2018), graph node classiﬁcation beneﬁts from low-frequency features.
That
GRU suppresses low-frequency band will result in a slight performance drop on the CLUSTER and
PATTERN datasets.
Ablation Study on High-Order GCNs. To contrast the performance gain produced by different orders on the benchmarks, we evaluate 1st-Order GCN, 3rd-Order GCN, 4th-Order GCN as well as their GRU variants on the ZINC, MNIST and CIFAR10 datasets.
Table 3 presents the results of our ablation study, which are consistent to our experiments on the synthetic datasets (Section 6.1).
As shown by our ablation study, aggregating zero-hop features brings about signiﬁcant improvements (Vanilla GCN vs. 1st-Order GCN), and adopting the second-hop features further promotes the performance (1st-Order GCN vs. SoGCN).
However, high-order GCNs are not capable of boosting the performance over SoGCN.
On the contrary, high-order GCs can even lead to the performance decline (3rd-Order GCN vs. 4th-Order GCN vs. SoGCN).
On the ZINC and MNIST datasets, we testify GRU’s effectiveness for each tested model, but the gain brought by GRU is not as signiﬁcant as aggregating the second-hop features.
On the CIFAR10 dataset, GRU fails to improve performance for 1st-Order GCN and 3rd-Order GCN.
C ONCLUSION
What should be the basic building blocks for GCNs?
To answer this, we seek the most localized graph convolution kernel (GC) with full expressiveness.
We generalize the ﬁlter space to a ﬁnite graph set and establish our LSS framework to assess GC layers functioning on different hops.
We show the second-order localized graph convolutional ﬁlter, called SoGC, possesses the full representation power than one-hop GCs.
Thus, it becomes the most localized GC that we adopt as the basic building block to establish our SoGCN.
Both synthetic and benchmark experiments exhibit the prominence of our theoretic design. We also make an empirical study on the GRU module cooperating with spectral GCNs.
Interesting directions for future work include analyzing two-hop aggregation schemes with message-passing GNNs and proving the universality of nonlinear GCNs.
A R EMARK ON D EFINITION
Let us rewrite the A of degree K in Deﬁnition 1:
[Equation]
which contains all LSI functions f : G × R N → R N with adjacency matrix as the graph shift.
We show this in the following way.
First, all linear transformations H invariant to the shift S should have HS =
SH .
Second, specifying arbitrary graph G ∈ G , any ﬁlter associated with it can be written as below:
[Equation]
where A ( G )
=
U Λ U T is the eigendecomposition of A ( G ) .
Then we can conclude the shiftinvariance property by the following Lemma 2.
Lemma 2.
Diagonalizable matrices
A 1 and A 2 are simultaneously diagonalized if and only if
A 1 A 2 =
A 2 A 1 .
B R ING I SOMORPHISM : A → T
We derive an equivalent form of A , namely construct a tractable space for A .
Notice that, this construction is essential to the proof of Lemma 1 and Theorem 1.
Since G is ﬁnite, we can construct a block diagonal matrix T ∈ R N |G|× N |G| , consisting of all adjacency matrices on the diagonal.
[Equation]
Hereby, we stop to explain the big picture of Deﬁnition 2 with Equation 15.
Obviously, the spectrum capacity Γ represents the number of eigenvalues of T without multiplicity.
Note that, eigenvalues of adjacency matrices signify graph similarity.
The spectrum capacity Γ identiﬁes a set of graphs by enumerating the structural patterns.
Even if the graph set goes extremely large (to guarantee the generalization capability), the distribution of spectrum provide the upper bound of Γ , so our theories will not lose generality.
Now we get back to construct a matrix space T from A via a ring homomorphism π
: A → T :
[Equation]
Recall that a ring homomorphism preserves the “summation” and “multiplication”.
Concretely, we write the matrix space T as follows:
[Equation]
In the following part, we prove that π is an isomorphism.
Proof.
First, the deﬁnition of T (Equation 17) basically conclude the surjectivity.
Second, for any f 1 (cid:54) = f 2 ∈
A with parameter α k , β k ∈ R where k = 0 , · · · , K , there exists G j ∈ G , x ∈ R N such that f 1 ( G j , x ) (cid:54) =
f 2 ( G j , x ) .
And we have their images H
= π ( f 1 ) , H
= π ( f 2 ) .
By padding x with zeros like:
[Equation]
where 0 N denote the all-zero vector of length N .
We apply H 1 , H 2 to x (cid:48) :
[Equation]
Hence, H 1 (cid:54) =
H 2 concludes the injectivity.
C P ROOF OF L EMMA
We ﬁrst show A is a vector space, then we leverage the isomorphism to have equality dim A = dim T .
Figuring out the dimension of T is much more tractable.
Proof.
By verifying the linear combination is closed or simply implied from the ring isomorphism π , A is at least a vector space
Then Lemma 1 follows from the Theorem 3 of Sandryhaila & Moura (2013).
We brieﬂy conclude the proof in the following way.
Let m ( x ) denote the minimal polynomial of T .
We have Γ =
deg m ( x ) .
Due to the isomorphism, dim A = dim T .
Suppose K + 1 < Γ .
First, dim T cannot be larger than K + 1 , because (cid:8)
I , T , · · · , T K (cid:9) is a spanning set.
If dim T < K + 1 , then there exists some polynomial p ( x ) with deg p ( x ) ≤ K , such that p ( A ) = 0 .
This contradicts the minimality of m ( x ) .
dim T can only be K + 1 .
Suppose K + 1 ≥ Γ .
For any H = h ( T ) for some polynomial h ( x ) with deg h ( x ) ≤ K .
By polynomial division, there exists unique polynomials q ( x ) and r ( x ) such that
[Equation]
where deg r ( x ) < deg m ( x ) =
Γ .
Insert T into Equation 21:
[Equation]
Therefore, (cid:8)
I , T , · · · , T Γ − 1 (cid:9) form a basis of T , i.e., dim T = Γ .
Remark that, we assume each graph contains the same number of vertices only for the sake of simplicity.
Lemma 1 still holds when the vertex numbers are varying, since the construction of Equation 15 is independent of this assumption.
However, we need the graph set to be ﬁnite, otherwise Γ might be uncountable.
We leave the discussion on inﬁnite graph sets for future study.
D P ROOF OF T HEOREM
First, we borrow the concept of T (Equation 17) in place of A .
Then we leverage the following basic yet powerful Lemma 3 to conclude the proof of Theorem 1 straightforwardly.
Lemma 3.
Over the ﬁeld of reals, the degree of an irreducible non-trivial univariate polynomial is either one or two.
Proof.
For any f ∈ A , let us map it to H
= h ( A ) through the isomorphism π
(Equation 16) for some polynomial h ( x ) with deg h ( x ) ≤ Γ − 1 (By Lemma 1).
By Lemma 3, factorize h ( x ) into series of polynomials with the degree at most two, and then merge ﬁrst-order polynomials into second-order ones until no paired ﬁrst-order polynomials remaining.
Finally, we obtain the following equation:
[Equation]
where D =
deg h ( x ) .
If D is even, deg h ( l ) = 2 for l = 1 , · · · , (cid:100) D/ 2 (cid:101) .
Otherwise, there exists some j ∈ [ L ] such that deg h
( j ) ( x )
= 1 .
Notice that, the remaining ﬁrst-order polynomials is at most 1, which indicates the sparsity of coefﬁcients is very low.
Now we obtain ﬁlters H ( l ) =
h ( l ) ( T ) for l = 1 , · · · , (cid:100) D/ 2 (cid:101) .
The last step is applying the inverse of isomorphism π − 1 to map H ( l ) ∈ T back to f ( l )
∈
A as below:
[Equation]
Recalling Section 4.1, f ( l ) ∈ B 2 for l = 1 , · · · , (cid:100) D/ 2 (cid:101) .
Since π − 1 is also a ring isomorphism,
H = H ( (cid:100) D/ 2 (cid:101) ) ◦ · · · ◦ H (1) implies f = f ( (cid:100) D/ 2 (cid:101) )
◦ · · · ◦ f (1) .
Whenever discussing the LSS in the spirit of Theorem 1, we refer to applying the ring isomorphism π and mapping the whole ﬁlter space to the matrix space T .
Using polynomials to analyze T is more straightforward.
E S YNTHETIC G RAPH S PECTRUM (SGS) D ATASET
Our SGS dataset works for node signal ﬁltering regression tasks.
We designed 3 types of graph signal ﬁlters: high-pass (HP), low-pass (LP) and band-pass (BP) ﬁlters, as given in Equation 25.
For each type, we generate 1000, 1000 and 2000 undirected graphs with graph signals and groundtruth response in the training set, validation set and test set, repectively.
Each graph approximately has 80 ˜120 nodes and 80 ˜350 edges.
Models trained on each sub-dataset are expected to learn the corresponding ﬁlter by supervising the MAE loss.
[Equation]
Undirected graphs are randomly sampled through rejection sampling of edges from complete graphs.
In detail, we randomly draw an integer N from [80 , 120] as the number of nodes, and then generate a
N × N random matrix B with each element independently sampled from Unif(0 , 1) .
Set a threshold (cid:15) and we can construct an adjacency matrix A by letting a i,j = 1 if b i,j > (cid:15) , otherwise a i,j = 0 , where a i,j is the element located at the i-th row and j-th column of A .
Next, we need to generate spectral signals s for the graph.
Independent sampling for each spectrum from a probabilistic distribution will only generate noises.
Hence, we synthesize spectrum by summing random functions.
We notice that pdf of the beta distribution Beta( a, b ) is a powerful tool to construct diverse curves by tuning shape parameters a and b .
Also, Gaussian function Norm( µ, σ ) is able to yield diverse bell-shaped curves by tuning µ and σ .
We sum 2 discretized beta functions, 4 discretized Gaussian functions with random parameters to generate spectral signals.
Equationelaborates the generation process and hyper-parameter choosing in our experiments, where g [ x ; a, b ] is the pdf of Beta( a, b ) distribution, f [ x ; µ, σ ] is the pdf of Norm( µ, σ ) distribution.
[Equation]
[Equation] [Equation]
] x
[Equation]
N ] a i , b i ∼ Unif
[Equation] [Equation]
µ
[Equation]
∼ Unif
[Equation]
0 , N
[Equation]
∼ Unif (cid:26)
N (
[Equation]
N j (cid:27)(cid:14)c j ∼
Unif { 0 . 5 , 2 } max
[Equation]
In most real-world cases, graph signals are usually represented in vertex-domain.
With a generated graph and its spectral signals, we can retrieve the vertex-domain signals by inverse graph Fourier transformation: perform eigen-decomposition on the normalized adjacency matrix
˜
A of the graph to retrieve its graph Fourier basis ˜ U , then we can ﬁnd vertex-domain signals by ˜ Us .
For supervising purpose, we retrieve the groundtruth of each ﬁlter’s response by applying each ﬁlter to the generated spectral signals, namely F ∗ k ( s ) , k ∈ { HP, LP, BP } .
Figure 4 illustrates an example of the generated spectral signals and its groundtruth ﬁltering response of the 3 ﬁlters.
[Figure]
F M ORE V ISUALIZATIONS OF S PECTRUM ON ZINC D ATASET
Each curve in the ﬁgure represents the spectrum of each graph signal channel.
We compute spectrum as follows: suppose we have an undirected and unweighted graph with normalized adjacency matrix
A and node signal X ∈ R N × D , where N is the number of nodes and D is the number of signal channels.
Since A is symmetric, we perform an eigen-decomposition on the adjacency matrix A =
U Λ U T .
Then the spectrum of X is computed by S
= U T X .
We plot each column of S as a curve.
More information about the graph spectrum and graph Fourier transformation can be found in Ortega et al.
[Figure]