R OBUST
G RAPH
R EPRESENTATION
L EARNING
VIA
N EURAL S PARSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph representation learning serves as the core of many important prediction tasks, ranging from product recommendation in online marketing to fraud detection in ﬁnancial domain
Real-life graphs are usually large with complex local neighborhood, where each node is described by a rich set of features and easily connects to dozens or even hundreds of neighbors.
Most existing graph learning techniques rely on neighborhood aggregation however the complexity on real- life graphs is usually high posing non-trivial overﬁtting risk during model train ing
In this paper, we present
Neural
Sparsiﬁcation (NeuralSparse), a supervised graph sparsiﬁcation technique that mitigates the overﬁtting risk by reducing the complexity of input graphs.
Our method takes both structural and non-structural information as input
, utilizes deep neural networks to parameterize the sparsi ﬁcation process and optimizes the parameters by feedback signals from down stream tasks.
Under the NeuralSparse framework, supervised graph sparsiﬁcation could seamlessly connect with existing graph neural networks for more robust performance on testing data.
Experimental results on both benchmark and private datasets show that,
NeuralSparse can effectively improve testing accuracy and bring up to 7.4% improvement when working with existing graph neural networks on node classiﬁcation tasks
I NTRODUCTION
Representation learning has been in the center of many machine learning tasks on graphs, such as name disambiguation in citation networks (Zhang et al., 2018c), spam detection in social networks (Akoglu et al., 2015), recommendations in online marketing (Ying et al., 2018a), and many others (Hamilton et al., 2017; Li et al., 2018).
As a class of models that can simultaneously utilize non-structural ( e.g. , node and edge features) and structural information in graphs, Graph Neural
Networks (GNNs) (Kipf & Welling, 2017; Hamilton et al., 2017; Li et al., 2016) construct effective representations for downstream tasks by iteratively aggregating neighborhood information (Kipf &
Welling, 2017; Hamilton et al., 2017).
Such methods have demonstrated state-of-the-art performance in classiﬁcation and prediction tasks on graph data (Veliˇckovi´c et al., 2018;
Chen et al., 2018;
Xu et al., 2019; Veliˇckovi´c et al., 2019).
Meanwhile, graphs from real-life applications are usually large with complex local neighborhood, where each node has rich features and dozens or even hundreds of neighbors.
As shown in Figure 1(a), this subgraph from Transaction dataset (detailed in Section 5.1) consists of 38 nodes ( i.e. , promising organizations and other organizations) with average node degree 15 and node feature dimension 120.
The GNNs are expected to grasp useful patterns from neighboring nodes; however, as representative patterns are diluted by overwhelming information in local neighborhood, graph learning algorithms could be misled by neighborhood aggregation.
Such complexity in input graphs poses non-trivial overﬁtting risk to existing GNN based learning techniques.
While it is straightforward yet expensive (sometimes even impractical) to address this overﬁtting problem by increasing the number of labeled samples, we investigate a cheaper alternative of reducing input graph complexity by graph sparsiﬁcation in this work.
Graph sparsiﬁcation (Liu et al., 2018; Zhang & Patone, 2017) aims to ﬁnd smaller subgraphs from input large graphs that best preserve desired properties.
Existing sparsiﬁcation methods could lead to suboptimal performance for downstream prediction tasks: (1) these methods are unsupervised such that the resulting sparsiﬁed
[Figure]
graphs may not favor downstream tasks; and (2) they only consider structural information for sparsiﬁcation decision, while non-structural information in graphs, such as node/edge features, could have non-trivial impact to the quality of sparsiﬁcation.
Recently, there have been GNN models attempting to sample subgraphs from predeﬁned distributions (Leskovec & Faloutsos, 2006; Adhikari et al., 2018; Hamilton et al., 2017; Chen et al., 2018).
As the predeﬁned distributions could be irrelevant to subsequent tasks, the sparsiﬁed graphs may miss important information for downstream tasks, leading to suboptimal prediction performance.
Present work .
We propose Neural Sparsiﬁcation (NeuralSparse), a general framework that simultaneously learns graph sparsiﬁcation and graph representation by feedback signals from downstream tasks.
The NeuralSparse consists of two major components: sparsiﬁcation network and GNN.
For the sparsiﬁcation network, we utilize a deep neural network to parameterize the sparsiﬁcation process: how to select edges from one-hop neighborhood given a ﬁxed budget.
In the training phase, the network learns to optimize a sparsiﬁcation strategy that favors downstream tasks.
In the testing phase, the network sparsiﬁes input graphs following the learned strategy, instead of sampling subgraphs from a predeﬁned distribution.
Unlike conventional sparsiﬁcation techniques, our technique takes both structural and non-structural information as input and optimizes the sparsiﬁcation strategy by feedback from downstream tasks, instead of using (possibly irrelevant) heuristics.
For the GNN component, the NeuralSparse feeds the sparsiﬁed graphs to a GNN and learns a graph representation for subsequent prediction tasks.
Under the framework of NeuralSparse, we are able to leverage the standard stochastic gradient descent and backpropagation techniques to simultaneously optimize graph sparsiﬁcation and representation.
As shown in Figure 1(b), the graph sparsiﬁed by the NeuralSparse has lower complexity with average node degree around 5.
As a result (illustrated in Figure 1(c)), the testing classiﬁcation accuracy on the sparsiﬁed graph is improved by 15%, compared with its counterpart in the original input graph, while conventional techniques could not offer competitive sparsiﬁcation for the classiﬁcation task.
Experimental results on both public and private datasets show that the NeuralSparse is able to consistently provide improved performance for existing GNNs on node classiﬁcation tasks, bringing up to 7% improvement.
R ELATED W ORK
Our work is related to two lines of research: graph sparsiﬁcation and graph representation learning.
Graph sparsiﬁcation .
The goal of graph sparsiﬁcation is to ﬁnd small subgraphs from input large graphs that best preserve desired properties.
Existing techniques are mainly unsupervised and deal with simple graphs without node/edge features for preserving predeﬁned graph metrics (H¨ubler et al., 2008), information propagation traces (Mathioudakis et al., 2011), graph spectrum (Calandriello et al., 2018; Chakeri et al., 2016; Adhikari et al., 2018), node degree distribution (Eden et al.,
[Figure]
𝑖 𝑗 𝑖 𝑗 𝑉𝑉𝐴′𝐴
GNN 𝑖 𝑗 𝑖 𝑗
MLP ; Voudigari et al., 2016), node distance distribution (Leskovec & Faloutsos, 2006), or clustering coefﬁcient (Maiya & Berger-Wolf, 2010).
Importance based edge sampling has also been studied in a scenario where we could predeﬁne edge importance (Zhao, 2015; Chen et al., 2018).
Unlike existing methods that mainly work with simple graphs without node/edge features in an unsupervised manner, our method takes node/edge features as parts of input and optimizes graph sparsiﬁcation by supervision signals from errors made in downstream tasks.
Graph representation learning .
Graph neural networks (GNNs) are the most popular techniques that enable vector representation learning for large graphs with complex node/edge features.
All existing GNNs share a common spirit: extracting local structural features by neighborhood aggregation.
Scarselli et al.
(2009) explore how to extract multi-hop features by iterative neighborhood aggregation.
Inspired by the success of convolutional neural networks, multiple studies (Defferrard et al., 2016; Bruna et al., 2014) investigate how to learn convolutional ﬁlters in the graph spectral domain under transductive settings (Zhang et al., 2018b; Zhuang & Ma, 2018).
To enable inductive learning, convolutional ﬁlters in the graph domain are proposed (Simonovsky & Komodakis, 2017; Niepert et al., 2016; Kipf & Welling, 2017;
Veli ˇ ckovi ´ c et al., 2018; Xu et al., 2018), and a few studies (Hamilton et al., 2017; Lee et al., 2018) explore how to differentiate neighborhood ﬁltering by sequential models.
In addition, multiple recent works (Ying et al., 2018b; Xu et al., 2019; Abu-ElHaija et al., 2019) investigate the expressive power of GNNs.
Recently, (Franceschi et al., 2019) study how to sample high-quality subgraphs from a space of all possible graphs of a complete graph so that the sampled graphs enhance the prediction power in downstream learning tasks.
In particular, the proposed method only focus on transductive tasks. Our work contributes from a unique angle: by reducing the noise from input graphs, our technique can further boost testing performance of existing GNNs.
P ROPOSED M ETHOD : N EURAL S PARSE
In this section, we introduce the core idea of our method.
We start with the notations that are frequently used in this paper.
We then describe the theoretical justiﬁcation behind NeuralSparse and our architecture to tackle the supervised node classiﬁcation problem.
Notations .
In this paper, we represent an input graph of n nodes as G =
( V, E, A ) : (1) V ∈ R n × d n includes node features with dimensionality d n ; (2) E ∈ R n × n is a binary matrix where E ( u, v )
=if there is an edge between node u and node v ; (3) A ∈ R n × n × d e encodes input edge features of dimensionality d e .
In addition, we use Y to denote the prediction target in downstream tasks ( e.g.,
Y ∈ R n × d l if we are dealing with a node classiﬁcation problem with d l classes).
Theoretical justiﬁcation .
From the perspective of statistical learning, the key of a deﬁned prediction task is to learn P ( Y | G ) , where Y is the prediction target and G is an input graph.
Instead of directly working with original graphs, we would like to leverage sparsiﬁed subgraphs to mitigate overﬁtting risks.
In other words, we are interested in the following variant,
[Equation]
where g is a sparsiﬁed subgraph, and S G is a class of sparsiﬁed subgraphs of G .
In general, because of the combinatorial complexity in graphs, it is intractable to enumerate all possible g as well as estimate the exact values of P ( Y | g ) and P ( g | G ) .
Therefore, we approximate the distributions by tractable functions,
[Equation]
where Q θ and Q φ are approximation functions for P ( Y | g ) and P ( g | G ) parameterized by θ and φ , respectively.
Moreover, to make the above graph sparsiﬁcation process differentiable, we employ reparameterization tricks (Jang et al., 2017) to make Q φ ( g | G ) directly generate differentiable samples, such that
[Equation]
where g (cid:48) ∼ Q φ ( g | G ) means g (cid:48) is a random sample drawn from Q φ ( g | G ) .
To this end, the key is how to ﬁnd appropriate approximation functions Q φ ( g | G ) and Q
θ ( Y | g ) .
Architecture .
In this paper, we propose Neural Sparsiﬁcation (NeuralSparse) to implement the theoretical framework discussed in Equation 3.
As shown in Figure 2, NeuralSparse consists of two major components: sparsiﬁcation network and GNNs.
• The sparsiﬁcation network is a multi-layer neural network that implements Q φ ( g | G ) :
Taking G as input, it generates a random sparsiﬁed subgraph of G drawn from a learned distribution.
• GNNs implement Q θ ( Y | g ) that takes a sparsiﬁed subgraph as input, extracts node representations, and makes predictions for downstream tasks.
Input: graph G while
As the sparsiﬁed subgraph samples are differentiable, the two components can be jointly trained using gradient descent based backpropagation techniques from a supervised loss function, as illustrated in Algorithm 1.
While the GNNs have been widely investigated in recent works (Kipf & Welling, 2017; Hamilton et al., 2017; Veliˇckovi´c et al., 2018), we focus on the practical implementation for sparsiﬁcation network in the remaining of this paper.
S PARSIFICATION N ETWORK
Following the theory discussed above, the goal of sparsiﬁcation network is to generate sparsiﬁed subgraphs for input graphs, serving as the approximation function Q φ ( g | G ) .
Therefore, we need to answer the following three questions in sparsiﬁcation network.
i ).
What is S G in Equation 1, the class of subgraphs we focus on? ii ).
How to sample sparsiﬁed subgraphs? iii ).
How to make sparsiﬁed subgraph sampling process differentiable for the end-to-end training?
In the following, we address the questions one by one.
k -neighbor subgraphs .
We focus on k -neighbor subgraphs for S G (Sadhanala et al., 2016):
Given an input graph, a k -neighbor subgraph shares the same set of nodes with the input graph, and each node in the subgraph can select no more than k edges from its one-hop neighborhood.
Although the concept of sparsiﬁcation network is not limited to a speciﬁc class of subgraphs, we choose kneighbor subgraphs for the following reasons.
• We are able to adjust the estimation on the amount of task-relevant graph data by tuning the hyper-parameter k .
Intuitively, when k is an under-estimate, the amount of task-relevant graph data accessed by GNNs could be inadequate, leading to inferior performance. When k is an overestimate, the downstream GNNs may overﬁt the introduced noise or irrelevant graph data, resulting in sub-optimal performance.
It could be difﬁcult to set a golden hyper-parameter that works all time, but one has the freedom to choose the k that is the best ﬁt for a speciﬁc task.
• k -neighbor subgraphs are friendly to parallel computation.
As each node selects its edges independently from its neighborhood, we can utilize tensor operations in existing deep learning frameworks, such as tensorﬂow (Abadi et al., 2016), to speed up the sparsiﬁcation process.
Sampling k -neighbor subgraphs .
Given k and an input graph G =
( V, E, A )
, we obtain a kneighbor subgraph by repeatedly sampling edges for each node in the original graph.
Without loss of generality, we sketch this sampling process by focusing on a speciﬁc node u in graph G .
Let N u be the set of one-hop neighbors of node u .
v ∼ f φ ( V ( u ) , V ( N u ) , A ( u )) , where f φ ( · ) is a function that generates a one-hop neighbor v from the learned distribution based on node u ’s attributes, node attributes of u ’s neighbors V ( N u ) , and their edge attributes A ( u ) .
In particular, the learned distribution is encoded by parameters φ . 2.
Edge E ( u, v ) is selected for node u . 3.
The above two steps are repeated k times.
Note that the above process performs sampling without replacement.
Given a node u , each of its adjacent edges is selected at most once.
Moreover, the sampling function f φ ( · ) is shared among nodes; therefore, the number of parameters φ is independent of the input graph size.
Making samples differentiable .
While conventional methods are able to generate discrete samples (Sadhanala et al., 2016), these samples are not differentiable such that it is difﬁcult to utilize them to optimize sample generation.
To make samples differentiable, we propose a Gumbel-Softmax based multi-layer neural network to implement the sampling function f φ ( · ) discussed in above.
To make the discussion self-contained, we brieﬂy discuss the idea of Gumbel-Softmax.
GumbelSoftmax is a reparameterization trick used to generate differentiable discrete samples (Jang et al., 2017; Maddison et al., 2017).
Under appropriate hyper-parameter settings, Gumbel-Softmax is able to generate continuous vectors that are as “sharp” as one-hot vectors widely used to encode discrete data.
Without loss of generality, we focus on a speciﬁc node u in a graph G =
( V, E, A ) .
Let N u be the set of one-hop neighbors of node u .
We implement f φ ( · ) as follows.
∀ v ∈ N
[Equation]
where MLP φ is a multi-layer neural network with parameters φ . 2.
∀ v ∈ N u , we employ a softmax function to compute the probability to sample the edge,
[Equation]
Using Gumbel-Softmax, we generate differentiable samples
[Equation]
where x u,v is a scalar, (cid:15) v = − log( − log( s )) with s randomly drawn from Uniform (0 , 1) and τ is a hyper-parameter called temperature which controls the interpolation between discrete distribution and continuous categorical densities.
Note that when we sample k edges, the computation for z u,v and π u,v only needs to be performed once.
For the hyper-parameter τ , we discuss how to tune it as follows.
Discussion on temperature tuning .
The behavior of Gumbel-Softmax is governed by a hyperparameter τ called temperature.
In general, when τ is small, the Gumbel-Softmax distribution resembles the discrete distribution, which induces strong sparsity; however, small τ also introduces high variance gradient that blocks effective backpropagation.
A high value of τ cannot produce expected sparsiﬁcation effect.
Following the practice in (Jang et al., 2017), we adopt the strategy by starting the training with a high temperature and anneal to a small value with a guided schedule.
Sparsiﬁcation algorithm and its complexity .
As shown in Algorithm 2, given hyper-parameter k , the sparsiﬁcation network visits each node’s one-hop neighbors k times.
Let m be the total number of edges in the graph.
The complexity of sampling subgraphs by the sparsiﬁcation network is O ( km ) .
When k is small in practice, the overall complexity is O ( m ) .
Input: graph G =
( V, E, A ) and integer k .
1: Edge set H = ∅ 2: for u ∈ V do 3: for v ∈ N u do 4: z u,v ← MLP φ ( V ( u ) , V ( v ) , A ( u, v )) 5: end for 6: for v ∈ N u do 7: π u,v ← exp( z u,v ) / (cid:80) w ∈ N u exp( z u,w ) 8: end for 9: for j = 1 , · · · , k do 10: for v ∈ N u do 11: x u,v ← exp((log( π u,v ) + (cid:15) v ) /τ ) / (cid:80) w ∈ N u exp((log( π u,w ) + (cid:15) w ) /τ ) 12: end for 13:
Add the edge represented by vector [ x u,v ] into H 14: end for 15: end for
Comparison with multiple related methods .
Unlike GraphSAGE (Hamilton et al., 2017), FastGCN (Chen et al., 2018), and AS-GCN (Huang et al., 2018) that incorporate layer-wise node samplers to reduce the complexity of GNNs, NeuralSparse samples subgraphs before applying GNNs.
As for the computation complexity, the sparsiﬁcation in NeuralSparse is more friendly to parallel computation than the layer-conditioned approach in AS-GCN.
Compared with GAT (Veli ˇ ckovi ´
c et al., 2018; Zhang et al., 2018a), the NeuralSparse can produce sparser neighborhood, which effectively mitigates overﬁtting risks.
Unlike LDS (Franceschi et al., 2019), NeuralSparse learns inductive graph sparsiﬁcation, and its graph sampling is constrained by input graph topology.
E XPERIMENTAL S TUDY
In this section, we evaluate our proposed NeuralSparse on node classiﬁcation task, including inductive and transductive settings.
We demonstrate that NeuralSparse achieves superior classiﬁcation performance over state-of-the-art GNN models.
Moreover, we provide a case study to demonstrate how sparsiﬁed subgraphs generated by NeuralSparse could improve classiﬁcation.
The supplementary material contains more detailed experimental information.
5. D ATASETS
We employ ﬁve datasets from various domains and conduct node classiﬁcation task following the settings as described in Hamilton et al.
(2017); Kipf & Welling (2017).
The dataset statistics are summarized in Table 1.
Inductive datasets.
We utilize the Reddit and PPI datasets and follow the same setting in Hamilton et al.
The Reddit dataset contains post-to-post graph with word vectors as node features.
The node labels represent which community Reddit posts belong to.
The protein-protein interaction (PPI) dataset contains graphs corresponding to different human tissues.
The node features are positional gene sets, motif gene sets and immunological signatures.
The nodes are multi-labeled by gene ontology.
The graph in the Transaction dataset contains real transactions between organizations in two years, with the ﬁrst year for training and the second year for validation/testing.
Each node [Table] represents an organization and each edge indicates a transaction between two organizations.
Node attributes are side information about the organizations such as account balance, cash reserve, etc.
On this dataset, we aim to classify organizations into two categories: promising or others for investment in near future.
The class distribution in the Transaction dataset is highly imbalanced.
During the training under inductive setting, algorithms have only access to training nodes’ attributes and edges.
In the PPI and Transaction datasets, the models have to generalize to completely unseen graphs.
Transductive datasets.
We use two citation benchmark datasets with transductive experimental setting in Yang et al.
; Kipf & Welling (2017).
The citation graphs contain nodes corresponding to documents and edges as citations.
Node features are the sparse bag-of-words representations of documents and node labels indicate the topic class of the documents.
In transductive learning, the training methods have access to all node features and edges, with a limited subset of node labels.
5. E XPERIMENTAL S ETUP
Baseline models .
We incorporate four state-of-the-art methods as the base GNN components, including GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2018), and GIN (Xu et al., 2019).
We evaluate our proposed NeuralSparse with sparsiﬁcation network and each of the four GNNs.
Besides, we also implement variants of NeuralSparse by replacing the sparsiﬁcation network with either the spectral sparsiﬁer (SS, Sadhanala et al., 2016) or the Rank
Degree (RD, Voudigari et al., 2016) method.
Temperature tuning.
We anneal the temperature with the schedule τ = max(0 .
05 , exp( − rp )) , where p is the training epoch and r ∈ 10 {− 5 , − 4 , − 3 , − 2 , − 1 } .
τ is updated every N steps and
N ∈ { 50 , 100 , ..., 500 } .
Compared with MNIST VAE model in Jang et al.
(2017), smaller hyperparameter τ ﬁts NeuralSparse better in practice.
Metrics.
We evaluate the performance on the transductive datasets with accuracy (Kipf & Welling, 2017).
For inductive tasks on the Reddit and PPI datasets, we report micro-averaged F1 scores (Hamilton et al., 2017).
Due to the highly imbalanced classes in the Transaction dataset, models are evaluated with AUC value (Huang & Ling, 2005).
The results show the average of 10 runs.
5. C LASSIFICATION P ERFORMANCE
Table 2 summarizes classiﬁcation performance of NeuralSparse and the baseline methods on all datasets.
For Reddit, PPI, Transaction, Cora and Citeseer, the hyper-parameter k is set as 30 , 15 , 10 , 5 , and 3 respectively.
The hyper-parameter l is set as 1 in this experiment.
Note that the result of
GAT on Reddit is missing due to the out-of-memory error.
Overall, NeuralSparse is able to help GNN techniques achieve competitive generalization performance with sparsiﬁed graph data.
We make the following observations.
Compared with basic GNN models, NeuralSparse can enhance the generalization performance on node classiﬁcation tasks by utilizing the sparsiﬁed subgraphs from sparsiﬁcation network, especially in the inductive setting.
Indeed, large neighborhood size in the original graphs could bring increased chance of introducing noise into the convolutional operations, leading to sub-optimal performance.
(2) With different GNN models, the NeuralSparse can consistently achieve comparable or superior performance, which demonstrates NeuralSparse is general and can be applied to multiple classiﬁcation models. [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
In comparison with the two NeuralSparse variants SS-GraphSAGE and RD-GraphSAGE, NeuralSparse outperforms because of the automatically learned graph sparsiﬁcation with both structural and non-structural information as input
[Figure]
5. S ENSITIVITY TO H YPER - PARAMETERS AND S PARSIFIED S UBGRAPHS
Figure 3(c) demonstrates how classiﬁcation performance responds when k increases on the Transaction dataset.
There exists an optimal k that delivers the best classiﬁcation AUC score.
When k is small, NeuralSparse can only make use of little relevant structural information in feature aggregation, which leads to inferior performance.
When k increases, the aggregation convolution involves more complex neighborhood aggregation with higher chance of overﬁtting noise data, which negatively impacts the classiﬁcation performance for unseen testing data.
Figure 3(d) shows how hyperparameter l impacts classiﬁcation performance on the Transaction dataset.
When l increases fromto 5 , we observe a relatively small improvement in classiﬁcation AUC score.
As the parameters in the sparsiﬁcation network are shared by all edges in the graph, the estimation variance from random sampling could already be mitigated to some extent by a number of sampled edges in a sparsiﬁed subgraph.
Thus, when we increase the number of sparsiﬁed subgraphs, the incremental gain could be small.
In Figure 3(a, b), we present the sparsiﬁed graphs output by two baseline methods, SS and RD.
By comparing the two plots with Figure 1(b), we make the following observations.
First, the NeuralSparse sparsiﬁed graph tends to select edges that connect nodes of identical labels, which favors the downstream classiﬁcation task.
The observed clustering effect could further boost the conﬁdence of decision making.
Second, instead of exploring all the neighbors, we can focus on selected connections/edges in sparsiﬁed graphs, which could make it easier for human experts to perform model interpretation and result visualization.
C ONCLUSION
In this paper, we propose Neural Sparsiﬁcation (NeuralSparse) to address the overﬁtting issues brought by the complexity in real-life large graphs.
NeuralSparse consists of two major components: (1) The sparsiﬁcation network sparsiﬁes input graphs by sampling edges following a learned distribution; (2) GNNs take sparsiﬁed subgraphs as input and extracts node representations for downstream tasks.
The two components in NeuralSparse can be jointly trained with supervised loss, gradient descent, and backpropagation techniques.
The experimental study on real-life datasets show that the NeuralSparse consistently renders more robust graph representations, and brings up to 7% improvement in accuracy over the state-of-the-art GNN models.
S D ATASET D ETAILS
In this section, we provide additional details about the Transaction datasets in our experiments.
The
Transaction dataset contains attributed graph that records transaction history between organizations in two years: 2014 and 2015.
Each node represents an organization and each directed edge indicates a transaction between two organizations.
Node attributes include organization information like account balance, cash research, etc.
Under the inductive experimental setting, We use the 47, organization data of the year 2014 for training and remaining data are hidden from the model.
The 9,554 organizations are used for validation and 38,218 for testing.
Validation and testing node sets are from year 2015 and are not connected to the nodes in the training set.
Like the PPI dataset, models need to generalize to unseen graph when testing on the Transaction dataset.
S E XPERIMENTAL S ETTINGS
In this section, we provide more details about our implementation and experiments in favor of reproducibility.
S2. H ARDWARE
All experiment are run on a Linux machine with 16 Intel(R)
Xeon(R) CPU (E5-2637 v4 @
3.50GHz) and 128GB RAM.
Some models (e.g. NeuralSparse and GCN) are accelerated by 4 NVIDIA
GeForce GTX1080Ti GPU with 11GB RAM.
S2. I MPLEMENTATIONS OF N EURAL S PARSE
We implement the proposed NeuralSparse in tensorﬂow framework for efﬁcient GPU computation.
In particular, the multi-layer neural network (Equation 4) in the sparsiﬁcation network is implemented by two-layer feed-forward neural networks in all experiment, where the hyper-parameter k is searched between 2 and 50 for the optimal performance.
We employ cross-entropy to formulate the loss function and apply Adam optimizer for training.
The learning rate of Adam optimizer is initially set to be α = 1 .
× 10 − 3 .
We initial the weight matrices in the proposed NeuralSparse model with Xavier initialization.
In the following, we detail the network structures of NeuralSparse used on individual datasets.
FC( a , b , f ) means a fully-connected layer with a input neurons and b output neurons activated by function f
(none means no activation function is used).
GNN( a , b , f ) means a Graph Neural Network layer with input dimension a , output dimension b , and activation function f .
We implement GNN layer with GCN, GraphSAGE, GAT, GIN in the experiments.
Reddit The sparsiﬁcation network runs with: FC(1204, 16, ReLU)-FC(16, 1, Gumbel-Softmax).
The structure of GNN is GNN(602, 128, ReLU)-GNN(128, 64, ReLU)-FC(64, 41, softmax).
PPI
The sparsiﬁcation network runs with: FC(100, 16, ReLU)-FC(16, 1, Gumbel-Softmax).
The structure of GNN is GNN(50, 128, ReLU)-GNN(128, 128, ReLU)-FC(128, 121, softmax).
Transaction The sparsiﬁcation network runs with: FC(243, 16, ReLU)-FC(16, 1, GumbelSoftmax).
The structure of GNN is GNN(121, 128, ReLU)-GNN(128, 32, ReLU)-FC(32, 2, softmax).
Note that there is one-dimensional edge attribute indicating the transaction amount in this dataset.
Cora The sparsiﬁcation network runs with: FC(2866, 32, ReLU)-FC(32, 1, Gumbel-Softmax).
The structure of GNN is GNN(1433, 128, ReLU)-GNN(128, 64, ReLU)-FC(64, 7, softmax).
Citeseer The sparsiﬁcation network runs with: FC(7406, 64, ReLU)-FC(64, 1, Gumbel-Softmax).
The structure of GNN is GNN(3703, 128, ReLU)-GNN(128, 64, ReLU)-FC(64, 6, softmax).
As the spectral sparsiﬁcation models cannot be jointly trained with subsequent GNN module, the sparsiﬁcation process is treated as a preprocessing step.
For Spectral Sparsiﬁer (SS), (cid:15) is set to 0 .in
all datasets.
For the Rank Degree algorithm (RD), we select 1% of nodes as the initial seeds and adopt ρ ∈ { 0 . 1 , 0 . 2 , · · · , 0 . 8 } for the best results.
S Q UALITATIVE E DGE S AMPLING E VALUATION
In this section, we qualitatively demonstrate the difference by Figure 1(a) original graph, Figure 1(b)
NeuralSparse, Figure 3(a) SS, and Figure 3(b) RD.
In addition, we provide quantitative analysis in Table S1, where we report the percentage of edges that connect nodes of same class labels in sparsiﬁed graphs.
Both qualitative and quantitative results suggest a common trend: NeuralSparse prefers to select neighbors with the same labels compared with the baseline methods. [Table] S E XPERIMENT WITH SIMILAR NUMBERS OF TRAINABLE PARAMETERS
In this section, we evaluate the impact brought by reducing the number of parameters in a GNN with
NeuralSparse so that the numbers of trainable parameters in a NeuralSparse GNN and an original
GNN are similar.
In particular, we focus on GCN in this set of experiment.
Using the same notation in S2.2, NeuralSparse-GCN-Compact is implemented as follows.
Reddit .
NeuralSparse-GCN-Compact runs with
: FC(1204, 8, ReLU)-FC(8, 1, Gumbel-Softmax) and GCN(602, 112,
ReLU
GCN(112, 64, ReLU)-FC(64, 41, softmax).
The total number of trainable parameters is 86,856 in the NeuralSparse
GCN-Compact, while it is 87,872 in the original
GCN
PPI .
NeuralSparse-GCN-Compact
runs with: FC(100, 16, ReLU)-FC(16, 1, Gumbel-Softmax) and
GCN(50, 118, ReLU)-GCN(118, 128, ReLU)-FC(128, 121, softmax).
The total number of trainable parameters is 38,108 in the NeuralSparse-GCN-Compact, while it is 38,272 in the original GCN.
Transaction .
NeuralSparse-GCN-Compact runs with: FC(243, 16, ReLU)-FC(16, 1, GumbelSoftmax) and GCN(121, 100, ReLU)-GCN(100, 32, ReLU)-FC(32, 2, softmax).
The total number of trainable parameters is 19,268 in the NeuralSparse-GCN-Compact, while it is 19,648 in the original GCN.
Cora NeuralSparse-GCN-Compact runs with: FC(2866, 8, ReLU)-FC(8, 1, Gumbel-Softmax) and
GCN(1433, 115, ReLU)-GCN(115, 32, ReLU)-FC(32, 7, softmax).
The total number of trainable parameters is 191,635 in the NeuralSparse-GCN-Compact, while it is 192,064 in the original GCN.
Citeseer NeuralSparse-GCN-Compact runs with: FC(7406, 32, ReLU)-FC(32, 1, Gumbel-Softmax) and GCN(3703, 64, ReLU)-GCN(64, 32, ReLU)-FC(32, 6, softmax).
The total number of trainable parameters is 476,256 in the NeuralSparse-GCN-Compact, while it is 482,560 in the original GCN.
[Table]
532 ± 0.
0.564 ± 0.
0.810 ± 0.
0.694 ± 0.
NeuralSparse0.946 ± 0.
0.600 ± 0.
0.610 ± 0.
0.821 ± 0.
0.715 ± 0.
GCN
NeuralSparse0.943 ± 0.
0.601 ± 0.
0.605 ± 0.
0.820 ± 0.
0.713 ± 0.
GCN-
Compact
From the evaluation results shown in Table S2, we draw the following observations.
First, both NeuralSparse-GCN and NeuralSparse-GCN-Compact consistently outperform GCN on all the datasets.
Second, compared with NeuralSparse-GCN, NeuralSparse-GCN-Compact achieves comparable prediction accuracy with smaller variance in most cases.
S
H OW PERFORMANCE EVOLVES AS HYPER - PARAMETER k CHANGES ?
[Figure]
In this section, we demonstrate how the hyper-parameter k impacts the performance of
NeuralSparse-GAT and NeuralSparse-GraphSAGE in both validation and testing on the Transaction dataset.
In terms of validation, as shown in Figure S1, the validation performance increases when k ranges from 2 to 10 with more available graph data.
After k exceeds 10, the increase in validation performance slows down and turns to be saturated.
In terms of testing performance, it shares a similar trend when k ranges from 2 to 10.
Meanwhile, the testing performance drops more after k exceeds 10.
S E MPIRICAL COMPARISON BETWEEN N EURAL S PARSE AND LDS S6. E VALUATION RESULTS
In this section, we compare NeuralSparse and LDS (Franceschi et al., 2019) with the datasets in transductive setting.
Here, we utilize three ways to prepare the input graphs of Cora and Citeseer datasets.
• Setting A: k-NN graphs (Franceschi et al., 2019).
In this setting, the graph structures are completely missing.
The input graphs are replaced with k-nearest neighbor graphs initialized from node features.
The k in kNN graph is selected from { 10, 20 } .
• Setting B: original input graphs of Cora and Citeseer datasets with the same random split as Kipf & Welling (2017).
• Setting C: edge union of original input graphs and k-NN graphs with k ﬁxed as 10.
[Table]
Our observation is summarized as follows.
In general, NeuralSparse and LDS achieves comparable node classiﬁcation accuracy.
Speciﬁcally, NeuralSparse has relatively better performance in Setting
[Table]
[Table]
A and Setting C. LDS performs slightly better in Setting B. From the above observation, we conjecture that NeuralSparse is more robust to graphs with more random edges while LDS is more suitable in a graph of relatively less noise by adding additional edges.
We will verify the conjecture in the next subsection.
S6. R ANDOM E DGE A DDITION TO C ORA AND C ITESEER
We further compare NeuralSparse and LDS (Franceschi et al., 2019) on the node classiﬁcation tasks where original graph structure is available but more random edges are introduced as noise.
Starting from the original graphs, we add edges by randomly sampling two nodes u , v from node set V and connecting them.
The results are shown in Figure S2.
In both datasets, NeuralSparse achieves better performance compared with LDS as the noise level goes beyond 200%.
When the amount of noise increases, the classiﬁcation accuracy of LDS drops signiﬁcantly.
This result conﬁrms our conjecture that NeuralSparse is more robust to random edges, compared to
LDS.
[Figure]
With the above comments and experimental results, we hope to clarify the difference between the two models and demonstrate that our proposed NeuralSparse is more robust to noises in real-life graphs.
S H OW DO TASK - IRRELEVANT EDGES COULD NEGATIVELY IMPACT THE PERFORMANCE OF GCN?
In this section, we use an example to demonstrate how an input graph with task-irrelevant edges could impact the performance of GCN.
For the ease of discussion and visualization, we focus on a GCN with a simple architecture and synthetic graphs where we could adjust graph topology by hyper-parameters in graph generators.
In terms of GCN, we investigate a one-layer GCN
[Equation]
where ˆ A is a normalized adjacency matrix, X is the input node feature matrix, W is the GCN parameters, and
Z =
ˆ AX denotes node representations in the aggregation space.
Intuitively, the quality of Z has direct impact to this GCN’s performance.
In terms of input graphs, we generate them for node classiﬁcation tasks as follows.
Nodes and their labels .
2,000 nodes are generated, where 1,000 nodes are assigned with positive labels and the rest are assigned with negative labels.
Node features .
Each node has a two-dimensional feature vector.
For positive nodes, the node features are generated from a Gaussian distribution, where µ pos =
( − 0 .
and Σ pos is a diagonal matrix with Σ pos [0 , 0]
= Σ pos [1 , 1] = 0 .
For negative nodes, the node features are generated from another Gaussian distribution, where µ neg = (0 . 5 , 0) and Σ neg is a diagonal matrix with Σ neg [0 , 0]
= Σ neg [1 , 1] = 0 .
Edges .
Given a hyper-parameter ¯ d , for each node, it randomly selects ¯ d nodes as its onehop neighbors.
With respect to this node classiﬁcation task, an edge that connects two nodes of different labels could be irrelevant, bringing noise to the GCN.
[Figure]
In Figure S3, the distributions of node representation Z are demonstrated at different ¯ d .
When ¯ d is 0, ˆ A is an identity matrix so that we simply use input node features in model learning.
As shown in
Figure S3(a), it is difﬁcult to ﬁnd a good boundary that well separates the positive and negative nodes by using node features only.
However, when we adjusts ¯ d to 10 or 20 with richer connections, the situation doesn’t get better.
Because of the noise introduced by irrelevant edges, it becomes harder to ﬁnd the classiﬁcation boundary.
While a deep learning may still be able to ﬁnd a complex boundary that well separates the training data, the boundary could overﬁt the introduced noise, resulting in low generalization power.
In Figure S4, we illustrate how NeuralSparse enhances the prediction accuracy of the GCN.
In particular, we focus on the graph with ¯ d = 20 , and sparsify this graph by NeuralSparse.
As shown in Figure S4, ranging the hyper-parameter k from 1 to 15, the distributions of Z vary.
When k is 1, there is no signiﬁcant change compared with the distribution in Figure S3(a), as the amount of accessible relevant graph data is still small.
When k is increased to 5 or 10 , the classiﬁcation boundary becomes much clearer.
As the edge generation process is uniformly random, the expected number of relevant edges per node is roughly 10.
When k is 15, this k could be an over-estimate on the amount of relevant edges, making it a bit harder to ﬁnd a good separation.
[Figure]
[Figure]
In Figure S5, we demonstrate how random downsampling could impact the prediction accuracy of the GCN.
In general, we could not see any signiﬁcant improvement.
Indeed, it is crucial to perform a task-driven sparsiﬁcation as NeuralSparse does.