A TTRIBUTED G RAPH L EARNING WITH 2-D G RAPH C ONVOLUTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph convolutional neural networks have demonstrated promising performance in attributed graph learning, thanks to the use of graph convolution that effectively combines graph structures and node features for learning node representations.
However, one intrinsic limitation of the commonly adopted 1-D graph convolution is that it only exploits graph connectivity for feature smoothing, which may lead to inferior performance on sparse and noisy real-world attributed networks.
To address this problem, we propose to explore relational information among node attributes to complement node relations for representation learning.
In particular, we propose to use 2-D graph convolution to jointly model the two kinds of relations and develop a computationally efﬁcient dimensionwise separable 2-D graph convolution (DSGC).
Theoretically, we show that DSGC can reduce intra-class variance of node features on both the node dimension and the attribute dimension to facilitate learning.
Empirically, we demonstrate that by incorporating attribute relations, DSGC achieves signiﬁcant performance gain over state-of-the-art methods on node classiﬁcation and clustering on several real-world attributed networks.
I NTRODUCTION
In an attributed graph, each node is associated with a feature vector, and nodes are connected by edges encoding their relations.
Commonly seen attributed graphs include citation networks where each node is a document represented by a bag-of-words feature vector and edges are citation links, web graphs where each webpage is also represented as a vector of words and edges are hyperlinks, social networks where each user is represented by a user proﬁle vector and edges indicate user friendship, and protein-protein interaction networks where each protein is represented by a list of protein signatures and edges encode interactions between proteins.
Learning on attributed graphs including node classiﬁcation and clustering ﬁnds many important applications in real-world networks.
Since the connectivity patterns and node contents of an attributed graph usually contain different information, it often requires joint modelling both aspects of information to achieve good learning performance.
A major class of methods (Yang et al., 2015; Pan et al., 2016;
Huang et al., 2017) is devoted to learning efﬁcient node representations of an attributed graph via nonnegative matrix factorization or random walk statistics and then perform downstream learning tasks with the learned representations.
A number of semi-supervised classiﬁcation methods (Belkin et al., 2006; Weston et al., 2008; Yang et al., 2016) classify nodes in an attributed graph by training a supervised classiﬁer on node features with some kind of graph regularizer.
Recently, a series of works based on graph convolutional neural networks including ChebyNet (Defferrard et al., 2016), graph convolutional networks (GCN) (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017) and graph attention networks (GAT) (Velickovic et al., 2018) have been shown to achieve state-of-the-art performance in node classiﬁcation and clustering (Kipf & Welling, 2016; Wang et al., 2017) on attributed graphs.
The key component of these models is one-dimensional (1-D) graph convolution, a function that naturally combines graph structures and node contents by aggregating a node’s features with its neighbours’.
As shown in (Li et al., 2019), the graph convolutional operator used in GCN and many follow-up works acts as a low-pass graph ﬁlter that smooths a node’s features with its neighbours’.
Under the assumption that nearby nodes tend to be in the same class, it can produce similar feature representations for nodes in the same class, thereby making them easier to be classiﬁed or clustered.
While this works well for attributed graphs with clear cluster structures, real-world networks could be highly noisy and sparse.
For example, in a web graph such as Wikipedia, a hyperlink between two webpages does not necessarily indicate that they belong to the same category, so mixing their features could be harmful for learning.
Furthermore, it has been shown that many real-world networks are scale-free (Albert & Barab´asi, 2002), which means there exist many low-degree nodes.
Since these nodes may have very few or even no links to other nodes, it would be difﬁcult and even impossible for them to learn similar feature representations as other same-class nodes, which is an intrinsic limitation of the 1-D graph convolution that is commonly adopted in existing models.
To address these limitations, we propose to explore relational information on a different dimension – the relations between feature attributes, in addition to node relations.
Likewise, a relation between two feature attributes should reﬂect some kind of similarity between them.
The assumption is that attributes that tend to indicate same classes should have strong relations.
For example, in a citation network, node attributes are words, and documents of AI category usually contain words such as “learning”, “robotics”, “machine”, “neural”, etc.
These indicative words for AI category should have much stronger relations among themselves than with other non-indicative words.
These informative relations can be used to construct an attribute afﬁnity graph to smooth node features in a similar way as the node relations do, only in a different dimension.
Importantly, attribute relations can complement node relations in node representation learning.
For instance, consider a document that has no links to others and hence it is impossible to do feature smoothing with node relations.
But with attribute relations, it can still learn similar feature representations as other same-class nodes.
In this paper, we make the following contributions.
• Methodology : We propose to use 2-D graph convolution to jointly model node relations and attribute relations for learning node representations of attributed graphs.
Further, we develop a computationally efﬁcient dimensionwise separable 2-D graph convolution (DSGC), which is equivalent to performing 1-D graph convolution alternately on the node dimension and the attribute dimension respectively.
• Theoretical Insight : We show that regular 1-D graph convolution on the node dimension can reduce intra-class variance of node features, which explains the success of many existing methods.
Further, we show that with a properly constructed attribute afﬁnity graph, graph convolution on the attribute dimension can also reduce intra-class variance of node features.
Jointly, our analysis provides a theoretical justiﬁcation of DSGC.
• Empirical Study : We implement DSGC for node classiﬁcation and clustering on attributed graphs, and compare it with state-of-the-art methods on a citation network, a web graph, and an email network.
The results demonstrate the superiority of DSGC over regular 1-D graph convolution on spare and noisy real-world attributed networks.
We also show that DSGC can be plugged into existing models to substantially improve their performance.
R ELATED W ORKS
Structural Graph Learning.
Methods for structural graph learning only utilize graph structures (node relations).
A common approach is to learn smooth low-dimensional embeddings of nodes using Markov random walks (Szummer & Jaakkola, 2002;
Perozzi et al., 2014; Grover & Leskovec, 2016), Laplacian eigenmaps (Belkin & Niyogi, 2004), spectral kernels (Chapelle et al., 2003; Zhang & Ando, 2006), autoencoders (Wang et al., 2016; Cao et al., 2016;
Ye et al., 2018) and generative adversarial nets (Dai et al., 2018; Wang et al., 2018).
Another direction is graph-based semi-supervised classiﬁcation, which includes methods based on low-density graph partition (Blum & Chawla, 2001;
Blum et al., 2004; Joachims, 2003) and the popular label propagation method and its variants (Zhu et al., 2003;
Zhou et al., 2004; Bengio et al., 2006; Hein & Maier, 2007; Wu et al., 2012).
Attributed Graph Learning.
Methods for attributed graph learning take into account both graph structures and node features.
A major class of methods learns node representations (Yang et al., 2015; Pan et al., 2016;
Huang et al., 2017) or clustering nodes (Xia et al., 2014; Zhou et al., 2010;
Li et al., 2018) by applying nonnegative matrix factorization, random walk statistics, or Laplacian eigenmaps on both graph structures and node features.
Some node clustering methods use a Bayesian model (Xu et al., 2012; Bojchevski & G¨unnemann, 2018) or design a distance measure that trades off structural and feature information (Zhou et al., 2009; Cheng et al., 2011).
Statistical relational learning methods model node relations and features with probabilistic graphical models, e.g., rela- tional Markov networks (Taskar et al., 2002).
Another category of related work is the graph-based semi-supervised node classiﬁcation methods that exploit both graph structures and node features.
For example, iterative classiﬁcation algorithm (Sen et al., 2008) iteratively classiﬁes an unlabeled node using its neighbours’ labels and features.
Manifold regularization (Belkin et al., 2006), deep semi-supervised embedding (Weston et al., 2008), and Planetoid (Yang et al., 2016) classify node features by training a supervised classiﬁer with a Laplacian or embedding-based regularizer.
Graph Neural Networks.
Another line of research on attributed graph learning is based on graph neural networks (Scarselli et al., 2009; Li et al., 2016).
Inspired by the success of convolutional neural networks (CNN) on Euclidean data, recent works (Bruna et al., 2014; Henaff et al., 2015;
Duvenaud et al., 2015; Atwood & Towsley, 2016) proposed to use 1-D graph convolution for attributed graph learning.
To avoid the expensive eigen-decomposition, ChebyNet (Defferrard et al., 2016) uses a polynomial ﬁlter represented by k -th order polynomials of graph Laplacian via Chebyshev expansion.
Graph convolutional networks (GCN) (Kipf & Welling, 2017) further simpliﬁes
ChebyNet by designing an efﬁcient layer-wise propagation rule via a ﬁrst-order approximation of spectral graph convolution.
GCN achieved outstanding results in semi-supervised classiﬁcation and inspired many follow-up works including MoNet (Monti et al., 2017), GraphSAGE (Hamilton et al., 2017), graph attention networks (Velickovic et al., 2018), gated attention networks (Zhang et al., 2018a), FastGCN (Chen et al., 2018b), dual graph convolutional neural network (Zhuang & Ma, 2018), stochastic GCN (Chen et al., 2018a), attributed network representation learning (Zhang et al., 2018c), LanczosNet (Liao et al., 2019), deep graph infomax (Velickovic et al., 2019), graph Markov neural networks (Qu et al., 2019), DisenGCN (Ma et al., 2019a), MixHop (Abu-El-Haija et al., 2019), etc. A recent interesting research direction is on learning graph structures for graph neural networks, e.g., Bayesian GCN (Zhang et al., 2018b) and LDS (Franceschi et al., 2019). Remarkably,
LDS can automatically learn a graph for data samples even when it is not available. Some attributed graph clustering methods based on 1-D graph convolution also showed promising performance, including graph autoencoder and graph variational autoencoder (Kipf & Welling, 2016), marginalized graph autoencoder (Wang et al., 2017), adversarially regularized graph autoencoder and adversarially regularized variational graph autoencoder (Pan et al., 2018), and adaptive graph convolution (Zhang et al., 2019).
More comprehensive reviews of graph neural networks can be found in (Cai et al., 2018;
Zhang et al., 2018d; Zhou et al., 2018).
2-D G RAPH C ONVOLUTION
In this section, we present 2-D graph convolution for attributed graph learning.
A comprehensive introduction of multi-dimensional graph Fourier transform can be found in (Kurokawa et al., 2017).
Different from Kurokawa et al.
(2017), here we propose a localized 2-D graph convolution to circumvent the computationally intensive graph Fourier transform.
Furthermore, we propose an even simpler dimensionwise separable 2-D graph convolution to efﬁciently model both node relations and attribute relations along the two dimensions of the feature matrix of an attributed graph.
3. 2-D G RAPH F OURIER T RANSFORM AND S PECTRAL G RAPH C ONVOLUTION
A 2-D graph signal is a function deﬁned on the Cartesian product of the vertex sets of two graphs.
Formally, given two weighted undirected graph G (1) and G (2) , denote the vertex sets by V (1) and
V (2) , the edge set by E (1) and E (2) , and the weighted adjacency matrix by A (1) and A (2) .
A 2D graph signal x on ( G (1) , G (2) ) is a real-valued function f : V (1) × V (2) → R , which can be conveniently represented in matrix form:
[Equation]
where n = |V (1) | and m =
|V (2) | .
In this paper, G (1) represents the given node graph; G (2) represents the constructed attribute afﬁnity graph; and X is the node feature matrix, which is a 2-D signal on the two graphs.
We call the adjacency matrices
A (1) and A (2) node afﬁnity matrix and attribute afﬁnity matrix respectively.
Deﬁne the graph Laplacian of G (1) and G (2) as L l = D (1) − A (1) and L r = D (2) − A (2) respectively.
Denote by λ i and µ j the eigenvalues of L l and L r , and U =
[ u 1 , · · · , u n ] and V =
[ v 1 , · · · , v m ] the corresponding eigenbasis respectively, then the n × m outer products
u i v (cid:62) j form a basis for the linear space R n × m .
It is known as 2-D graph Fourier basis – an analogy of the Fourier basis in classical harmonic analysis in graph domain.
The corresponding eigenvalue pair ( λ i , µ j ) is known as the frequency of basis matrix u i v (cid:62) j .
Then, a 2-D graph signal X can be decomposed as:
[Equation]
Then, we can deﬁne 2-D graph Fourier transform as S = U (cid:62) XV , where S is called the spectrum of signal X or Fourier coefﬁcients.
Based on 2-D graph Fourier transform, we can now manipulate 2-D graph signals in the spectral (frequency) domain and deﬁne 2-D spectral graph convolution.
By the convolution theorem, the convolution of two signals equals to point-wise multiplication of their spectrum in the spectral domain.
2-D spectral graph convolution is a function conv : R n × m → R n × m that takes signal X as input and outputs a new signal Z :
[Equation]
where p ( λ, µ ) : R × R → R is the frequency response of the convolution; ◦ is Hadamard (elementwise) product; and P ∈ R n × m with P ij = p
( λ i , µ j ) is the frequency response in matrix form.
3. F AST L OCALIZED 2-D S PATIAL G RAPH C ONVOLUTION
Although Eq.
well deﬁnes 2-D graph convolution, it is often impractical to perform convolution in the spectral domain, due to the high cost of computing the eigenbasis U , V needed for Fourier transform.
Similar to (Defferrard et al., 2016) on 1-D graph convolution, here we propose 2-D spatial graph convolution to avoid intensive computation.
Without loss of generality, we restrict the frequency response p ( · , · ) to be a polynomial of two variables λ, µ with parameters Θ ∈ R n × m , i.e., p ( λ, µ ) =
(cid:80) i,j θ
ij
λ
i µ j .
Then, the 2-D spectral graph convolution in Eq.
(3) becomes
[Equation]
Eq.
(4) is called 2-D spatial graph convolution , as it manipulates the signal X in the spatial domain.
Parameter Θ is called the kernel of the convolution.
Here, the spatial convolutional ﬁlter is localized.
Denote by k 1 and k 2 the largest exponent of λ and µ in the polynomial p respectively, then i > kand j > k 2 imply θ ij = 0 .
The convoluted signal z ij of vertex pair ( ν (1) i , ν (2) j ) only depends on the neighbourhood of ν (1) i within k 1 hops and the neighbourhood of ν (2)
j within k 2 hops, so the ﬁlter is said to be k 1 -localized on G (1) and k 2 -localized on G (2) , and the size of the kernel Θ is k 1 × k 2 .
Dimensionwise Separable 2-D Graph Convolution (DSGC)
Although the above spatial graph convolution avoids the computationally expensive Fourier transform, its general form with kernel size k 1 × k 2 still involves at least k 1 × k 2 matrix multiplications.
Inspired by the depthwise separable convolution proposed in (Howard et al., 2017), we streamline spatial graph convolution by restricting the rank of Θ to be one.
Consequently, Θ is decomposed as an outer product of two vectors θ (1) ∈ R n and θ (2) ∈ R m .
The frequency response p can be decomposed as a product of two single variable polynomials, i.e., p ( λ, µ ) =
p 1 ( λ ) p 2 ( µ ) , where p 1 ( λ ) = (cid:80) i θ (1)
i λ i and p 2 ( µ ) =
(cid:80)
j θ (2) j µ j .
Finally, the 2-D spatial graph convolution in Eq.
(4) becomes
Z = GXF , where G = p 1 ( L l ) and F = p 2 ( L r ) .
We call Eq.
(5) dimensionwise separable graph convolution (DSGC).
The fastest way to compute it only requires k 1 + k 2 matrix multiplications, much less than the k 1 × k 2 matrix multiplications needed by a general 2-D spatial graph convolution.
We call GX node graph convolution and G the node graph convolutional ﬁlter.
Similarly, we call
XF attribute graph convolution and F the attribute graph convolutional ﬁlter.
Notably, G and F can complement each other to learn better node representations.
As illustrated in Fig. 1, the node afﬁnity Att1 Att2 Att3 Att4 ab c d e a b c d e a 2/3 1/ b 1/3 1/3 1/ c 1/3 2/ d e
Att1 Att2 Att3 Att4 a 2/3 1/ b 1/3 2/ c
d e
Att1 Att2 Att3 Att4 a 1/2 1/ b 1/2 1/ c 1/2 1/ d 1/2 1/ e 1/2 1/2 Att1 Att
Att3 Att4 Att1 1/2 1/
Att2 1/2 1/
Att 1/2 1/ Att 1/2 1/
Node of Class Node of Class
Attribute
[Figure]
GXF
F
Att4 Att
Att
Att
Attribute affinity graph
[Figure]
graph only captures node relations of Class 1, and applying node graph convolution on the feature matrix X smooths the node features of Class 1, but it does not affect those of Class 2.
Fortunately, the attribute afﬁnity graph encodes complementary relational information on another dimension, and further applying attribute graph convolution makes the node features of Class 2 similar and those of
Class 1 even more similar, thereby making them much easier to be classiﬁed or clustered.
I NTRA - CLASS V ARIANCE R EDUCTION BY DSGC
Given a data distribution, the lowest possible error rate an classiﬁer can achieve is the Bayes error rate (Fukunaga, 2013), which is caused by the intrinsic overlap between different classes and cannot be avoided.
In this section, we show that DSGC with proper graph convolutional ﬁlters can reduce intra-class variance of the data distribution while keeping class centers roughly unchanged, hence reducing the overlap between classes and improving learning performance.
Intra-class Variance and Inter-class Variance.
Suppose samples x i and their labels
y
i are observations of a random vector X =
[ X 1 , · · · , X m ] (cid:62) and a random variable Y respectively.
We deﬁne the variance of random vector X to be the sum of the variance of each dimension X j , i.e., the trace of the covariance matrix of X .
According to law of total variance (Grinstead & Snell, 2012), the variance of X can be divided into intra-class variance and inter-class variance:
[Equation]
where the conditional variance Var ( X | Y = k ) is the variance of class k and the conditional expectation E
[ X | Y = k ] is the k -th class center.
Intra-class variance (IntraVar) measures the average divergence within each class, while inter-class variance (InterVar) measures the divergence among class centers.
We are interested in the IntraVar / InterVar ratio.
Here, we assume that each class of data Pr
( X | Y = k ) follows a unimodal distribution (e.g. Gaussian, chi-square, Laplace), a class of most common distributions in the real world, and with roughly convex contours.
Under this assumption, a low IntraVar / InterVar ratio generally indicates low classiﬁcation error, since the class overlap is reduced.
This is also veriﬁed by our experiments on real attributed networks in section 7.
4. I NTRA - CLASS V ARIANCE R EDUCTION BY N ODE G RAPH C ONVOLUTION
As variance of sample mean is always less than variance of individual samples, averaging samples of the same class can always reduce intra-class variance.
Actually this is how node graph convolution works.
For any node i , node graph convolution GX produces a new feature vector z i = (cid:80) j G ij x j .
When G is a stochastic matrix, the output feature vector z i is a weighted average of the neighbours of x i .
Denote by Z a random vector of z i .
Intuitively, as long as each node i has enough sameclass neighbours, Z will have a smaller IntraVar / InterVar ratio than X .
Formally, assume that the graph is generated by a stochastic block model, where nodes from the same class are connected with probability p and those from different classes are connected with probability q .
Then, with the stochastic graph ﬁlter G = D − 1 A (1) , we have the following theorem.
Theorem 1.
When q is sufﬁciently small, the IntraVar / InterVar ratio of Z is less than or equal to that of X , i.e.,
[Equation]
The proof is given in the Appendices.
Under the assumption that nodes in the same class are most likely to be connected, node graph convolution G can reduce intra-class variance while keeping inter-class variance roughly unchanged, thereby decreasing the IntraVar/InterVar ratio.
4. I NTRA - CLASS V ARIANCE R EDUCTION BY A TTRIBUTE G RAPH C ONVOLUTION
In the following, we show that a proper attribute graph convolutional ﬁlter F can also reduce the
IntraVar/InterVar ratio.
We use the convention that the random vector X is a column vector, and hence the attribute graph convolution XF results in a new random vector F (cid:62) X .
We also assume that the node features are mean-centered, i.e. E [ X ] = 0 .
Denote by Cov ( k )
=
Cov ( X | Y
=
k ) the covariance matrix of X w.r.t.
each class, and denote by π k
= Pr( Y = k ) the portion of each class.
Then, after attribute graph convolution, the intra-class variance of data becomes
[Equation]
Theorem 2.
If the attribute graph convolutional ﬁlter F is a doubly stochastic matrix, then the output of attribute graph convolution has an intra-class variance less than or equal to that of X , i.e., (cid:88) [Equation] 1 and F ij ≥ 0 , ∀ i, j [Equation]
Var
[Equation]
The proofs of Eq.
(8) and Theorem 2 are given in the Appendices.
Eq.
(8) suggests that to reduce intra-class variance, the attribute afﬁnity graph should connect attributes X i and X j with small or even negative covariance Cov ( k ) ij .
For example, in Figure 1, the attributes Att3 and Att4 are both indicative of class 2, but negatively correlated w.r.t.
class 2, so connecting them in the attribute afﬁnity graph can greatly reduce the intra-class variance of class 2.
To achieve a low IntraVar/InterVar ratio, in addition to reducing intra-class variance, we also need to keep the class centers apart after convolution, which then depends on the quality of the attribute afﬁnity graph.
A good attribute afﬁnity graph should connect attributes that share similar expectations conditioned on Y .
Formally, each attribute X j has K conditional expectations w.r.t.
Y , which are denoted as a vector e j = ( E [ X j | Y = 1] , · · · , E [ X j | Y = K ]) ∈ R K .
We have the following.
Theorem 3.
If ∀ F ij (cid:54) = 0 , (cid:107) e
i − e j (cid:107) 2 ≤ ε , then the distance between e j and (cid:98) e j = (cid:80) i F ij e i is also less than or equal to ε , i.e., [Equation]
[Equation]
and ε can be arbitrarily small with a proper F .
By Theorem 3, since the conditional expectations of each attribute may change little after attribute graph convolution, one can infer that the class centers will also change little, and so does the interclass variance.
Combining Theorems 2 & 3, it suggests that a proper attribute afﬁnity graph should connect attributes that have similar class means but are less positively correlated, so as to achieve a low IntraVar / InterVar ratio and improve performance.
Again, it can be seen in Figure 1 that the attributes Att3 and Att4 have exactly the same class means, but are negatively correlated w.r.t.
class 2, and connecting them in the attribute afﬁnity graph reduces the intra-class variance of class 2 to 0.
A TTRIBUTED G RAPH L EARNING WITH DSGC 5. U NSUPERVISED N ODE R EPRESENTATION L EARNING
Given an attributed graph with node feature matrix X , we can learn the node representations Z in an unsupervised manner by applying DSGC on X , i.e.,
[Equation] and then perform various downstream learning tasks with Z .
Node Classiﬁcation.
With the learned node representations Z , we can simply train a classiﬁer such as multilayer perceptron with the labeled nodes, and then apply the trained classiﬁer on the unlabeled nodes to predict their labels.
Node Clustering.
With the learned node representations Z , we propose to cluster nodes as follows.
We ﬁrst apply a linear kernel on Z to learn pairwise proximity between nodes, i.e., K = ZZ (cid:62) , and then perform spectral clustering (Perona & Freeman, 1998; Von Luxburg, 2007) on K .
5. E ND - TO -E ND S EMI -S UPERVISED L EARNING
For semi-supervised learning on attributed graphs, we can also use DSGC to replace the 1-D graph convolution used in existing end-to-end semi-supervised learning models including GCN (Kipf &
Welling, 2017), GAT (Velickovic et al., 2018) and GraphSAGE (Hamilton et al., 2017).
For example, to incorporate DSGC into the vanilla GCN, we can modify the ﬁrst layer propagation of GCN as:
[Equation]
where H (1) is the hidden units in the ﬁrst layer, W (
1) ∈
R m × l is the trainable parameters of GCN, and σ is a nonlinear function such as ReLU.
Both G and F are ﬁxed ﬁlters, where G is the node graph convolutional ﬁlter of GCN, and F is the proposed attribute graph convolutional ﬁlter.
Importantly, Eq.
(10) can be considered as feeding a ﬁltered feature matrix XF instead of the raw feature matrix X to GCN.
By our above analysis, a proper attribute graph ﬁlter F can reduce intra-class variance, which makes XF much easier to classify and guarantees to help train a better model.
Further, it can be shown that the model trained by Eq.
(10) is essentially different from that of GCN almost surely.
For GCN, the model is freely chosen from the parameter space W (1) , while the model trained by Eq.
(10) is restricted in a subspace F W (1) .
Since F is low-pass (section 6.2), it is also low-rank, and F W (1) is a small subspace of R m × l projected by F .
Model parameters in this subspace are generally better in terms of the generalization performance (test accuracy), due to the variance reduction property of F .
However, the model learned by Eq.
(10) can hardly be learned by
GCN, since the subspace F W (1) has measure zero, which is an extremely small subset of R m × l .
I MPLEMENTATION OF DSGC 6. N ODE G RAPH C ONVOLUTIONAL F ILTERS
The node afﬁnity graph ( A (1) ) is given as part of the dataset.
There are various graph convolutional ﬁlters available (Li et al., 2019), e.g., the one used in GCN (Kipf & Welling, 2017) is a symmetrically normalized node afﬁnity matrix.
In our experiments, we use a row normalized node afﬁnity matrix (consistent to our analysis in section 4.1) of order 2 (following GCN) as the ﬁlter for node graph convolution: [Equation]
[Equation]
where D 1 is the degree matrix of A (1) .
We observe in our experiments that the performance of the symmetrically normalized node afﬁnity matrix (GCN ﬁlter) is similar.
6. A TTRIBUTE G RAPH C ONVOLUTIONAL F ILTERS
A key issue in implementing DSGC is to construct the attribute afﬁnity graph ( A (2) ).
Possible ways of constructing A (2) include extracting entity relation information from existing knowledge bases, building a similarity graph from features, or identifying correlations by domain knowledge.
In the following, we describe two methods for text data, which have been proven useful by our experiments.
Positive point-wise mutual information (PPMI).
Positive PMI (Church & Hanks, 1990) is a common tool for measuring the association between two words in natural language processing.
Given a pair of words w i and w j , the edge weight is deﬁned as the PPMI between w i and w j :
[Equation]
where p ( w i , w j ) and p ( w i ) are learned by sliding a window over a large corpus of text.
PMI reﬂects word collocation, as it assumes that if two words co-occur more than expected under independence, there must be some kind of semantic relation between them, which is often true in practice.
With the constructed attribute afﬁnity graph A (2) by PPMI, in our experiments, we use a symmetrically normalized afﬁnity matrix as the ﬁlter:
[Equation]
(13) where D 2 is the degree matrix of A (2) .
Word embedding based k -NN graphs.
Word embedding is a collection of techniques that map vocabularies to vectors in a Euclidean space.
Embeddings of words are pre-trained vectors learned from corpus with algorithms such as GloVe (Pennington et al., 2014).
Since word embeddings capture semantic relations between words (Bakarov, 2018), they can be used for constructing an attribute afﬁnity graph.
With the embedding vectors, we can construct a k -NN graph with some proximity metric such as the Euclidean distance.
With the constructed attribute afﬁnity graph A (2) , in our experiments, we use the following one-step lazy random walk ﬁlter (Li et al., 2019):
[Equation]
14) where D 2 is the degree matrix of A (2) .
Note that the ﬁlters in Eq.
(13) and (14) are slightly different, since the diagonal weight of the afﬁnity matrix constructed by word embedding is much smaller than that of the PPMI afﬁnity matrix, and we want to assign sufﬁcient weight to each attribute itself.
Also, we use ﬁlters of order 1 since the PPMI afﬁnity matrix is very dense.
According to the analysis in
Li et al.
(2019), Eq.
(13) and (14) are both low-pass ﬁlters.
E MPIRICAL S TUDY
To validate the effectiveness of DSGC, we conduct extensive experiments for semi-supervised node classiﬁcation and node clustering on three large real-world attributed networks including 20 Newsgroups (20 NG) (Lang, 1995),
Wikispeedia (Wiki) (West et al., 2009; West & Leskovec, 2012) and
Large Cora (L-Cora) (McCallumzy et al., 1999;
Li et al., 2019).
1 Due to space limitation, details of datasets, experimental setup and computational time are provided in the Appendices.
Variance Reduction.
First of all, to verify our analysis in section 4, we demonstrate the variance reduction effect of both node graph convolution and attribute graph convolution.
As shown in Figure 2, 1-D graph convolution with either G or F already greatly reduces the IntraVar / InterVar ratio, and together they further signiﬁcantly reduce the ratio.
7. N ODE C LASSIFICATION
Baselines.
We test the proposed node classiﬁcation method by DSGC with or without using the node graph convolutional ﬁlter ( G ) and the attribute graph convolutional ﬁlter ( F ) in ﬁve cases.
We use a two-layer multilayer perceptron as the classiﬁer for DSGC.
PPMI and Emb denote constructing the attribute afﬁnity graph with positive point-wise mutual information and word embedding respectively.
We compare DSGC with the following baselines: label propagation (LP) (Wu et al., 2012), multi-layer perceptron (MLP), GCN (Kipf & Welling, 2017), generalized label propagation (GLP)
(Li et al., 2019), GraphSAGE (Hamilton et al., 2017), graph attention networks (GAT) (Velickovic et al., 2018), and deep graph infomax (DGI) (Velickovic et al., 2019).
We also try to incorporate
DSGC into GCN, GAT, GraphSAGE, and LDS (Franceschi et al., 2019) (see the experimental setup of LDS in Appendix A.4) as described in section 5.2 to improve their performance.
Performance.
We test semi-supervised node classiﬁcation under two scenarios – 20 labels per class and 5 labels per class.
For each task, we report the mean classiﬁcation accuracies and standard deviations over 10 runs, as summarized in Table 1, where the top 2 accuracies are highlighted in bold.
In Table 2, we report the results of GAT, GCN, GraphSAGE, and LDS after incorporating DSGC.
The following observations can be made
[Table]
[Figure]
[Table]
[Figure] [Figure] [Table] 1) Node graph convolution does not always help.
On 20 NG and L-Cora, methods based on node graph convolution such as DSGC ( GX ), GLP, GCN and GAT all outperform MLP signiﬁcantly.
However, on Wiki, node [Table]
convolution harms the performance, which is due to the highly noisy hyperlink graph with intra-class edge ratio 38.0%, much lower than that of 20 NG (96.8%) and LCora (76.5%) (see Table 5 in the Appendices).
This shows the limitation of node graph convolution.
Attribute graph convolution works.
On all the three datasets, DSGC ( XF ) outperforms MLP by a large margin.
This shows that attribute graph convolution can learn useful node representations even when there are no connections between nodes. Remarkably, on Wiki, where the hyperlink graph is of bad quality, DSGC ( XF ) with either PPMI or Emb outperforms all the baselines by a large margin. DSGC ( XF ) with PPMI improves upon the best baseline by 4% and 10% in absolute accuracy for tasks with 20 labels and 5 labels per class respectively.
3) 2-D graph convolution is effective.
• For datasets with good node graphs such as 20 NG and L-Cora, DSGC ( GXF ) performs much better than either DSGC ( GX ) or DSGC ( XF ).
DSGC ( GXF ) with PPMI achieves the best performance. On 20 NG, it improves upon the best baseline by more than 5% for tasks withlabels per class, and about 4% for tasks with 5 labels per class, both in absolute accuracy. On LCora, it is comparable with the best baseline. On both datasets, for tasks with 20 labels per class, DSGC ( GXF ) with Emb also outperforms all the baselines; for tasks with 5 labels per class, it outperforms all the baselines except DGI.
This shows that node graph convolution and attribute graph convolution can complement each other and lead to signiﬁcant performance gain. • For datasets with bad node graphs such as Wiki, DSGC ( GXF ) signiﬁcantly improves upon DSGC ( GX ) and outperforms all the baselines except GraphSAGE, which uses extra labeled data for validation while DSGC uses none. Nevertheless, since node graph convolution degrades performance on such a noisy graph, DSGC ( XF ) performs better than DSGC ( GXF ). [Table]
• As shown in Table 2, for node classiﬁcation with 20 labels per class on all three datasets, after incorporating DSGC, the performances of GCN, GAT, LDS, and GraphSAGE are substantially improved in most cases. This further veriﬁes the effectiveness of 2-D graph convolution.
7. N ODE C LUSTERING
Baselines.
We test the proposed node clustering method by DSGC (section 5.1) with or without using G and F in ﬁve cases.
We compare DSGC with strong baselines for attributed graph clustering including GAE and VGAE (Kipf & Welling, 2016), MGAE (Wang et al., 2017), ARGE and ARVGE (Pan et al., 2018), and AGC (Zhang et al., 2019).
We also compare with spectral clustering (Spectral) that operates on a similarity matrix constructed by applying linear kernel on the node representations.
Performance.
To evaluate the clustering performance, we adopt two widely-used metrics (Aggarwal & Reddy, 2014): clustering accuracy (Acc) and normalized mutual information (NMI).
The results are shown in Table 3, where the top 2 results are highlighted in bold.
The following observations can be made.
1) Attribute graph convolution works very well.
On 20 NG, DSGC ( XF ) with PPMI already outperforms most baselines by a large margin.
On Wiki, DSGC ( XF ) with PPMI or Emb signiﬁcantly outperforms all the baselines.
2) Similar to the classiﬁcation tasks, 2-D graph convolution is effective.
On 20 NG, DSGC ( GXF ) with PPMI can further improve upon the very strong performance of DSGC ( XF ) and performs the best; On L-Cora, DSGC ( GXF ) with PPMI or Emb improves upon either DSGC ( GX ) or DSGC ( XF ) and outperforms most baselines signiﬁcantly.
On Wiki, DSGC ( XF ) performs better than DSGC ( GXF ), due to the low-quality hyperlink graph as explained above.
For both classiﬁcation and clustering, we observe that in most cases DSGC with PPMI can achieve better performance than with Emb.
This shows the effectiveness of PPMI in capturing meaningful word relations based on information theory and statistics (Church & Hanks, 1989), whereas Emb only relies on a distance metric for measuring word similarity.
C ONCLUSION
In this paper, we have proposed to model attributed graphs with 2-D graph convolution.
We have demonstrated theoretically and empirically that by exploiting attribute relations in addition to node relations, a simple and efﬁcient dimensionwise separable 2-D graph convolution (DSGC) can learn better node representations than existing methods based on regular 1-D graph convolution on noisy and sparse real-word networks.
We believe 2-D graph convolution is a promising tool for attributed graph learning, and there is much left to be explored.
In future work, we plan to further investigate the construction of attribute afﬁnity graphs for different types of attributed networks, apply DSGC to solve various practical problems, and design new efﬁcient ﬁlters for 2-D graph convolution.
Appendices
Appendix A provides details about the experiments in section 7 and presents additional experimental results.
Appendix B, C and D provide proofs of the theorems in section 4.
A PPENDIX A E XPERIMENT D ETAILS AND S UPPLEMENTARY E XPERIMENTS A. D ATASET D ETAILS
[Table]
20 Newsgroups (20 NG) Lang (1995) is an email discussion group, where each node is an email and there are 18846 emails in total.
Each email is represented by an 11697-dimension tf-idf feature vector.
Two emails are connected by an edge if they replies the same one.
These emails are categorized into 20 classes as listed in Table 4.
Wikispeedia (Wiki) West et al.
(2009); West & Leskovec (2012) is a webpage network in which the nodes are 3767 Wikipedia webpages, and the edges are web hyperlinks.
Each webpage is described by a 18316-dimension tf-idf vector.
We remove several tiny classes from the dataset, so the webpages distribute more evenly across the remaining 9 categories, which are also listed in Table 4.
Large Cora (L-Cora) McCallumzy et al.
(1999) is a citation network in which the nodes are computer science research papers represented by 3780 dimension of tf-idf values.
Two papers are connected by an undirected edge if and only if one cites the other.
These citation links form a node graph.
The nodes in the dataset are originally categorized into a topic hierarchical tree with 73 leaves.
After removing the papers that belong to no topic and the ones that have no authors or title, a subset of 11881 papers is obtained Sacc´a et al.
These papers are then classiﬁed into 10 highest-level topics in the topic hierarchy, as listed in Table 4.
We name this dataset “Large Cora” to distinguish it from the “Cora” dataset with 2708 papers used in Kipf & Welling (2017);
Yang et al.
Sen et al.
Note that we did not test on this ”Cora” and the “Citeseer” and “PubMed” datasets as in Kipf & Welling (2017); Yang et al.
Sen et al.
(2008), because the attributes (words) are not provided in these datasets.
The statistics of the datasets are summarized in Table 5, where the last row shows the intra-class edge ratio of the node graph of each dataset, which can reﬂect the quality of the graph.
A. V ISUALIZATION
In Figure 3, we visualize the results of performing graph convolution on the node features of 20 NG dataset by t-SNE.
It can be seen that graph convolution can successfully reduce the overlap among classes, and 2-D graph convolution is more effective than 1-D.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
A. T RAINING T IME
The training time is summarized in Figure 4.
We can see that DSGC is several times faster than
GCN, GAT, DGI and GraphSAGE.
This is because DSGC only performs graph convolution once before training, while others need to do graph convolution in every training step.
[Figure]
[Figure]
For DSGC, we test positive point-wise mutual information (PPMI) and word embedding based k -NN graphs (Emb) for constructing the attribute afﬁnity graph.
For constructing the PPMI graph, we set the context window size to 20 and use the inverse distance as co-occurrence weight.
For constructing the Emb graph, we use GloVe Pennington et al.
(2014) to learn word embeddings and set the number of nearest neighbours k = 20 .
The classiﬁer of DSGC is a two-layer multi-layer perceptron (MLP) with 64 hidden units.
The MLP is trained with Adam Optimizer for 200 epochs with 0.015 learning rate.
For the sake of fair comparison to all baselines and to make sure our method can work well in practice, we do not use extra labeled data for validation on our methods and the baselines, unless otherwise stated.
Instead, we always select the model with the lowest train loss in the 200 epochs.
In each run, the dataset is randomly split into a labeled set and an unlabeled set.
The classiﬁcation results are averaged over 10 runs.
Hyperparameters of all models, including our methods and baselines, are tuned to its best performance by grid search.
DSGC follows the network setting and hyperparameters of GCN, which are a 2-layer structure with 64 hidden units and 0.015 learning rate.
We set the balancing parameter α in
LP as 100.
We largely follow Velickovic et al.
(2018) to set network structures and hyperparameters of GAT, which are 2 layers, 8 heads, 8 neurons of each head, 0.005 learning rate, 400 epochs, and 0.
dropout.
Our setting of DGI is the same as Velickovic et al.
(2019), except that the number of hidden units is decreased to 128,
so it can be ﬁtted into our GPU memory.
Our setting of GraphSAGE is the same as Hamilton et al.
(2017) (using 30% data as validation set), except that the number of samples in layer 2 is reduced to 5, also due to limited GPU memory.
Parameters of GLP are same as Li et al.
For LDS Franceschi et al.
(2019), since it cannot scale to the size of 20 NG (out of GPU memory), we follow the authors to select a 10-category subset of 20 NG.
For each dataset, both the training set and validation set for LDS contain 20 labels per class.
All models are tested on a platform with Intel(R)
Xeon(R) CPU E5-2640 v4 @ 2.40GHz and single NVIDIA(R) GeForce(R)
GTX 1080 Ti.
A. P ARAMETER S ETTINGS FOR N ODE C LUSTERING
For our method DSGC, the attribute afﬁnity graphs PPMI and Emb are constructed in the same way as in node classiﬁcation.
For other baselines, we follow the parameter settings described in the original papers.
In particular, for GAE and VGAE (Kipf & Welling, 2016), we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer, and train the encoders foriterations using the Adam algorithm with a learning rate of 0.01.
For MGAE (Wang et al., 2017), the corruption level p is 0.4, the number of layers is 3, and the parameter λ is 10 − 5 .
For ARGE and
ARVGE (Pan et al., 2018), we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer.
The discriminators are built by two hidden layers with 16 neurons and 64 neurons respectively.
We train all the autoencoder-related models for 200 iterations and optimize them using the Adam algorithm.
The learning rate and the discriminator learning rate are both 0.001.
For AGC (Zhang et al., 2019), the maximum iteration number is 60.
For fair comparison, these baselines also adopt spectral clustering as DSGC to get the clustering results.
We repeat each method for 10 times, and show the mean clustering results and the standard deviations.
A PPENDIX B P ROOF OF T HEOREM 1 IN S ECTION 4.
We suppose that nodes from the same class are connected with probability p , and nodes from different classes are connected with probability q , i.e., the adjacency matrix A (1) of node graph obeys
[Equation]
and G = D − 1 A (1) is a stochastic matrix.
We also assumes that classes are balanced, i.e., Pr( Y = k )
= 1 /K for all k , then we have following theorem.
Theorem 1.
When q is sufﬁciently small, Z has an IntraVar / InterVar ratio less than or equal to that of X , i.e.,
[Equation]
Proof.
The proof consists of two parts.
In the ﬁrst part, we prove that inter-class variance is unchanged after node graph convolution, when q approximates 0, i.e.,
[Equation]
In the second part, we prove that intra-class variance becomes smaller after node graph convolution, i.e.,
[Equation]
18) when G is a stochastic matrix.
Part 1.
Inter-class variance is unchanged.
Since
[Equation]
we have
[Equation]
# by linearity of expectation
[Equation]
When q approximates 0 , E [ z i | y i
= k ] will approximate E
[ X | Y =
k ] , so
E [ Z | Y =
k ] = (cid:88)
i,y i =
k
Pr( Z =
z
i | y i
= k ) E
[ z i | y i
=
k ] = (cid:88)
i,y i =
k
Pr( Z =
z
i | y i
= k ) E
[ X | Y =
k ] =
E [ X | Y =
k ] .
Hence,
Var ( E [ Z | Y ])
=
Var ( E [ X | Y ]) .
Part 2.
Intra-class variance becomes smaller.
Denote by Cov ( · , · ) the covariance of two random variables.
We have
Var
[Equation]
# property of variance
[Equation]
# property of covariance =
[Equation]
Since
Var ( z i | y i = k ) =
Var 
[Equation]

 ≤ 
[Equation]
j
G ij (cid:113) Var ( x
[Equation]
y
[Equation]
= k ) 

[Equation] # by the above inequality =  (cid:88)
j
G ij (cid:112) Var (
[Equation]
) 
 =
(cid:16)(cid:112) Var ( X | Y = k ) (cid:17)
[Equation]
# since (cid:88)
j
G ij ==
Var ( X | Y = k ) , we have
Var ( Z | Y = k ) =
(cid:88) i,y i = k
Pr( Z =
z
i | y i =
k )
Var ( z i | y i = k )
≤ Var ( X | Y = k ) .
Then we have
E [ Var ( Z | Y )]
= (cid:88) k
Pr( Y = k ) Var ( Z | Y = k ) ≤ (cid:88) k
Pr( Y = k ) Var ( X | Y
[Equation]
# by the above inequality
[Equation] [Equation]
Combining Eq.
(19) and Eq.
, we prove that when q is sufﬁciently small, [Equation]
[Equation]
A PPENDIX C P ROOF OF T HEOREM 2 IN S ECTION 4.
Theorem 2.
If the attribute graph convolutional ﬁlter F is a doubly stochastic matrix, then the output of attribute graph convolution has an intra-class variance less than or equal to that of X , i.e., (cid:88)
i
F ij = (cid:88)
j
F ij = 1 and F ij ≥ 0 , ∀ i, j ⇒
E (cid:2)
Var (cid:0) F (cid:62) X
| Y (cid:1)(cid:3)
≤ E [ Var ( X | Y )] .
Proof.
We ﬁrst prove a lemma that variance of each class will not increase after attribute graph convolution, i.e., Var (cid:0) F (cid:62) X | Y =
k (cid:1)
≤ Var ( X | Y = k ) .
Denote by Cov ( · ) the covariance matrix of a random vector.
Based on our deﬁnition of variance in section 4, we have
Var
[Equation]
=
[Equation]
Y
[Equation]
# property of covariance =
[Equation]
# cyclic property of trace =
[Equation]
# property of trace
[Equation]
# property of covariance
[Equation]
# eigenvalues of F is no more than
[Equation]
Next
[Equation]
we prove the theorem with the above lemma.
Denote by π k =
Pr( Y = k ) the portion of each class, then we have
[Equation]
Construction of a Doubly Stochastic Filter F
Given an attribute afﬁnity matrix A (2) , one could easily construct a doubly stochastic ﬁlter F by the following steps: 1) compute graph Laplacian
L (2) = D (2) − A (2) ; 2) compute doubly stochastic graph matrix Ω
= ( I + L (2) )
− 1 ; 3) take Ω as a new attribute afﬁnity matrix, choose a polynomial p with nonnegative coefﬁcients that sum to 1, and let F = p ( Ω ) .
It can be easily seen that F is doubly stochastic.
A PPENDIX D P ROOF OF T HEOREM 3 IN S ECTION 4.
Theorem 3.
If ∀ F ij (cid:54) = 0 , (cid:107) e
i − e j (cid:107) 2 ≤ ε , then the distance between e j and (cid:98) e j = (cid:80) i F ij e i is also less than or equal to ε , i.e., (cid:107) e
i − e j (cid:107)
[Equation]
≤
[Equation]
, ∀ F ij (cid:54) =
⇒ (cid:107)
[Equation]
and ε can be arbitrarily small with a proper F .
Proof.
[Equation]
# since (cid:88)
i
F ij =
[Equation]
# Cauchy-Schwarz inequality
[Equation]
Next, we prove that there exists such an F that ε is 0.
This is equivalent to ﬁnding a doubly stochastic
F satisfying (cid:80) i F
ij e i =
e j for all j .
Since F = I is a trivial solution, it is solvable.
Denote by m the number of attributes, and denote by K the number of classes.
This linear system consists of m ( K + 2) equations and m 2 variables.
In most real-world attributed networks, the number of attributes is far greater than the number of classes, so the number of variables in this linear system is greater than the number of equations.
Given that it is solvable, it must have inﬁnite number of solutions other than I .
Thus, ε can be arbitrarily small with a proper F .