D EEP G ENERATIVE M ODELS FOR G ENERATING L ABELED G RAPHS
A BSTRACT
As a new way to train generative models, generative adversarial networks (GANs) have achieved considerable success in image generation, and this framework has also recently been applied to data with graph structures.
We identify the drawbacks of existing deep frameworks for generating graphs, and we propose labeledgraph generative adversarial networks (LGGAN) to train deep generative models for graph-structured data with node labels.
We test the approach with different discriminative models as well as different GAN frameworks on various types of graph datasets, such as collections of citation networks and protein graphs.
Experiment results show that our model can generate diverse labeled graphs that match the structural characteristics of the training data and outperforms all baselines in terms of quality, generality, and scalability.
I NTRODUCTION
Graphs are powerful complex data structures that can describe collections of related objects.
Such collections could be atoms forming molecular graphs, users connecting on online social networks, and papers connected by citations.
The connected objects, or nodes, may be of different types or classes.
Methods that reason about this ﬂexible and rich representation can empower analyses of important, complex real-world phenomena.
It is important to be able to learn generative models for the distributions of graphs, which can encode knowledge about the nature of the objects represented by graphs.
In this paper, we introduce a method that learns generative models for labeled graphs in which the nodes and the graphs may have categorical labels.
Recently Goodfellow et al.
(2014) described generative adversarial networks (GANs), which have been widely explored in computer vision and natural language processing (Zhang et al., 2017; Yu et al., 2017) for generating realistic images and text, as well as performing tasks such as style transfer.
GANs are composed of two neural networks.
The ﬁrst is a generator network that learns to map from a latent space to the distribution of the target data, and the second is a discriminator network that tries to distinguish real data from candidates synthesized by the generator.
Those two networks compete with each other during training and each improve based feedback from the other.
The success of this general GAN framework has proven it to be a powerful tool for learning the distributions of complex data.
Motivated by the power of GANs, researchers have used it for generating graphs too.
Bojchevski et al.
(2018) proposed NetGAN, which uses the GAN framework to generate random walks on graphs.
De Cao & Kipf (2018) proposed MolGAN, which generate molecular graphs using the combination of a GAN framework and a reinforcement learning objective.
However, there are many limitations of existing methods, such as the generality to graphs with different structures and scalability to different sized graphs.
Furthermore, they are unable to generate graphs with node labels, a critical feature of some graph-structured data.
The rapid development of deep learning techniques has also led to advances in representation learning in graphs.
Many works have been proposed to use deep learning structures to extract high-level features from nodes and their neighborhoods to include both node and structure information (Kipf & Welling, 2017; Hamilton et al., 2017; Fan & Huang, 2017).
These methods have been shown to be useful for many applications, such as link prediction and collective classiﬁcation.
Among all these works, Xu et al.
(2018) propose jumping knowledge networks for learning adaptive, structureaware representations that are particularly useful for representation learning on complex graphs with diverse graph structures.
Building on these advances, we propose labeled graph generative adversarial network (LGGAN), a deep generative model trained using a GAN framework to generate graph-structured data with node labels.
LGGAN can be used to generate various kinds of graph-structured data, such as citation graphs, knowledge graphs, and protein graphs.
Speciﬁcally, the generator in an LGGAN generates an adjacency matrix as well as labels for the nodes, and its discriminator uses a jumping knowledge network (Xu et al., 2018) to identify real graphs using adaptive, structure-aware higher-level graph features.
Our approach is the ﬁrst deep generative method that addresses the generation of labeled graph-structured data.
In experiments, we demonstrate that our model can generate realistic graphs that preserve important properties from the training graphs.
We evaluate our model on various datasets with different graph types—such as ego networks and proteins—and with different sizes.
Our experiments demonstrate that LGGAN effectively learns distributions of different graph structures and that it can scale up to generate large graphs without losing much quality.
R ELATED W ORK
Generative graph models were pioneered by Erd ¨ os & R ´ enyi (1959), who introduced random graphs where each possible edge appears with a ﬁxed independent probability.
More realistic models followed, such as the preferential attachment model of (Albert & Barab´asi, 2002), which grows graphs by adding nodes and connecting them to existing nodes with probability proportional to their current degrees.
Goldenberg et al.
(2010) proposed the stochastic block model (SBM), and Airoldi et al.
(2008) proposed the mixed-membership stochastic block model (MMSB).
The SBM is a more complex version of the Erd¨os-R´enyi (E-R) model that can generate graphs with multiple communities.
In SBMs, instead of assuming that each pair of nodes has identical probability to connect, they predeﬁne the number of communities in the generated graph and have a probability matrix of connections among different types of nodes.
Compared to the E-R model, SBMs are more useful since they can learn more nuanced distributions of graphs from data.
However, SBMs are still limited in that they can only generate graphs with this kind of community structure.
With the recent development of deep learning, some works have proposed deep models to represent the distribution of graphs.
Li et al.
(2018) proposed DeepGMG, which introduced a framework based on graph neural networks.
They generate graphs by expansion, adding new structures at each step.
Li et al.
(2018) proposed GraphRNN, which decomposes the graph generation into generating node and edge sequences from a hierarchical recurrent neural network.
Simultaneously, researchers have also been developing other implicit yet powerful methods for generating graphs, especially based on the success of generative adversarial networks (Goodfellow et al., 2014).
For example, Bojchevski et al.
(2018) proposed NetGAN, which uses the GAN framework to generate random walks on graphs from which structure can be inferred, and De Cao & Kipf (2018) proposed MolGAN to generate molecular graphs using the combination of the GAN framework and reinforcement learning.
However, these recently proposed deep models are either limited to learning from a single graph (Kipf & Welling, 2016;
Grover et al., 2018;
Bojchevski et al., 2018), to generating small graphs with 40 or fewer nodes (Li et al., 2018), or to generating speciﬁc types of graphs such as molecular graphs (You et al., 2018a; De Cao & Kipf, 2018) (with no straightforward generalization to other domains due to specialized tools to calculate molecule-speciﬁc loss).
Most broadly, most of these recently proposed methods cannot be used to generate labeled graphs.
M ODEL
In this section, we introduce LGGAN and how it trains deep generative models for graph-structured data with node labels.
3. A RCHITECTURE
In this section, we provide details on the LGGAN architecture.
As in the standard GAN framework,
LGGAN consists of two main components: a generator G and a discriminator D .
The generator G takes a sample from a prior distribution and generates a labeled graph g represented by an adjacency matrix A and a node label matrix L .
The discriminator D then trains to distinguish samples from the dataset and samples from the generator.
In LGGAN, both the generator and the discriminator are trained using the improved Wasserstein GAN (Gulrajani et al., 2017) approach.
Generator LGGAN’s generative model uses a multi-layer perceptron (MLP) to produce the graph.
The generator G takes a random vector z sampled from a standard normal distribution and outputs two matrices: (1) L ∈ R N × C , which is a one-hot vector that deﬁnes the node labels; and (2) the adjacency matrix A ∈ R N × N , which deﬁnes the connections among nodes in graphs.
The architecture uses a ﬁxed maximum number of nodes N , but it is capable of generating structures of fewer nodes.
[Table]
Discriminator
The discriminator D takes a graph sample as input (represented by an adjacency matrix A and a node label matrix L ) and outputs a scalar value (and also a one-hot vector for class label if using the AC-GAN framework).
For the discriminator
[Table]
we propose two models, a simple one and an advanced discriminative model.
The simple model is just an MLP while the advanced model uses the jumping knowledge networks (Xu et al., 2018)
(JK-Net).
The JK-Net is comprised by a series of graph convolution layers and a layer aggregation operator to integrate useful information from each layer for learning more powerful graph representations.
We refer to the whole framework using the simple model as LGGAN s and using the JK-Net as simply LGGAN.
JK-Nets with layer aggregation have been shown to extract more adaptive and structure-aware representations compared to other representation learning methods such as graph convolutional networks (GCN) (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017) and graph attention networks (GAT)
(Veli ˇ ckovi ´ c et al., 2018).
For our JK-Net, we use a GCN as the base model, and it propagates based on the following rule:
[Equation]
where H ( l ) ∈ R N × D is output matrix at the l − 1 th layer, I N is the identity matrix, ˜
A =
A + I N is the adjacency matrix of the graph g with self-connections added,
˜ D ii
=
(cid:80)
j ˜
A ij is the diagonal degree matrix of graph g , W
( l ) ∈ R D
× F is a the trainable weight matrix at l th layer, and σ ( · ) denotes an activation function (such as the sigmoid or ReLU (Nair & Hinton, 2010)).
Since we do not include node attributes, we set H (0) =
I N where I N is the identity matrix.
After n layers of propagation via graph convolutions, we aggregate the outputs from each layer with an aggregation function agg, such as concatenation and max-pooling.
We then concatenate the aggregated matrix with the node label matrix L and outputs Z g as the ﬁnal representation we learned for the graph g :
[Equation]
[Equation]
The representation Z g of the graph will further be processed by a linear layer to produce the outputs of the discriminator: the graph-level scalar probability of the input being real data.
The structure of this discriminative model is showing in Figure 2.
If we are using the AC-GAN framework, there will also be a classiﬁer to predict the category that the graph belongs to with a one-hot vector c .
3. A LTERNATE GAN FRAMEWORKS
Since the GAN framework was introduced by Goodfellow et al.
(2014), many variations have been proposed that proved to be powerful for generation tasks.
Therefore, we adopt three popular variations and compare how well they perform for our task of labeled graph generation.
We use the traditional, original GAN approach as the ﬁrst approach.
Beyond the traditional GAN framework, we use two other methods that include extra information of classiﬁcation labels for the graphs themselves.
The ﬁrst follows the conditional GAN (Mirza & Osindero, 2014) framework, which feeds the graph label as an extra input to the generator in addition to the noise z .
We can use this label to generate the graphs of different types.
To improve on this, our last variation uses the auxiliary conditional GAN (Odena et al., 2017) framework, in which the discriminator not only distinguishes whether the graph is real or fake, but it also incorporates a classiﬁer of the graph labels.
The structure of all those three variations is illustrated in Figure 1.
[Table]
3. T RAINING
GANs (Goodfellow et al., 2014) train via a min-max game with two players competing to improve themselves.
In theory, the method converges when it reaches a Nash equilibrium, where the samples produced by the generator matches the data distribution.
However, this process is highly unstable and often results in problems such as mode collapse (Goodfellow, 2016).
To deal with the most common problems in training GAN, such as mode collapse and unstable training, we use the Wasserstein
GAN (Salimans et al., 2016) with a gradient penalty.
We also adopt several techniques such as feature matching and minibatch discrimination that were shown to encourage convergence and help avoid mode collapse.
3. N ODE O RDERING
A common representation for graph structure uses adjacency matrices.
However, using matrices to train a generative model introduces the issue of how to deﬁne the node ordering in the adjacency matrix.
There are n !
permutations of n nodes, and it is time consuming to train over all of them.
For LGGAN, we use the framework of JK-Net with graph convolutions (Kipf & Welling, 2017) and a node aggregation operator (De Cao & Kipf, 2018) as the discriminator.
This discriminator is invariant to node ordering, avoiding the issue.
However our simpliﬁed version, LGGAN s, uses a
MLP as the discriminator, which is dependent on node ordering.
Therefore, in this case, we arrange the nodes in a breadth-ﬁrst-search (BFS) ordering for the training data.
We adopt this strategy from
You et al.
(2018b), who generated graphs by adding nodes in a BFS ordering.
In particular, we preprocess the adjacency matrix A and node label matrix L by feeding them into a
BFS function.
This function takes a random permutation π g of the nodes in graph g as input, picks a node v i as the starting node, and then outputs another permutation π (cid:48)
g that is a BFS ordering of the node in graph g starting from node v i .
In this way, by specifying the node ordering in the graph, we only need to train on all possible BFS orderings, rather than all possible node permutations.
This reduction makes a huge difference for computational complexity when graphs are large.
E XPERIMENTS
In this section, we ﬁrst explore the six different variations of LGGAN (three different GAN frameworks for either LGGAN or LGGAN s) and discuss their advantages and disadvantages.
We also compare LGGAN with other graph generation methods to demonstrate its ability to generate highquality labeled graphs in diverse settings.
4. B ASELINES
We compare our model against various traditional generative models for graphs, as well as some recently proposed deep graph generative models.
For traditional baselines, we compare against the Erd¨os-R´enyi model (E-R) (Erd¨os & R´enyi, 1959), the Barab´asi-Albert (B-A) model (Albert & Barab´asi, 2002), and mixed-membership stochastic block models (MMSB) (Airoldi et al., 2008).
Then we also compare with some recently proposed deep graph generative models such as the DeepGMG (Li et al., 2018) and GraphRNN (You et al., 2018b).
Few current approaches are designed to generate labeled graphs.
One exception is MolGAN (De Cao & Kipf, 2018), which is designed to generate molecular graphs and needs specialized evaluation methods speciﬁc to that task, so we do not compare against it.
We also do not compare with NetGAN (Bojchevski et al., 2018) since its framework is constrained to learn to generate a single new graph from a single training graph.
4. D ATASETS
We perform experiments on different kinds of datasets with varying sizes and characteristics, described in the following.
Citation graphs We test on scientiﬁc citation networks.
We used the Cora and Citeseer datasets (Sen et al., 2008).
The Cora dataset is a collection of 2,708 machine learning publications categorized into seven classes, and the CiteSeer dataset is a collection of 3,312 research publications crawled from the CiteSeer repository.
To test the scalability of LGGAN, we extracted different subsets with different graph sizes by constraining the number of nodes in graph | V | .
For small datasets (denoted cora small and citeseer small), we extract two-hop and three-hop ego networks with 30 ≤ | V | ≤ 50 .
For the large datasets (denoted cora and citeseer), we extract three-hop ego networks with 150 ≤ | V | ≤ 200 .
When using the AC-GAN framework, we set the graph label to be the node label of the center node of the ego network.
Protein graphs We also test on multiple collections of protein molecular graphs.
The ENZYMES dataset consists of 600 enzymes (Schomburg et al., 2004).
Each enzyme in the dataset is labeled with one of the six enzyme commission (EC) code top-level classes.
The protein dataset includes proteins from the dataset of enzymes and non-enzymes created by Dobson & Doig (2003).
There are two graph labels: enzymes and non-enzymes.
4. E VALUATION
To evaluate the quality of the generated graphs, we follow the approach used by You et al.
(2018b): we compare a distribution of generated graphs with that of real ones by measuring the maximum mean discrepancy (MMD) (Gretton et al., 2012) of graph statistics, capturing how close their distributions are.
We use four graph statistics to evaluate the generated graphs: degree distribution, clustering coefﬁcient distribution, node-label distribution, and average orbit count statistics.
Since we are generating labeled graphs, we also want to evaluate the graph distribution in each class.
To do this, we extract subgraphs for each class from both the training graphs and generated graphs and evaluate based on these three metrics (excluding the label distribution).
These per-label tests help test whether the model simply assigns the class based on the label distribution without considering the underlying graph structure.
Since these label-based evaluation metrics are only applicable to labeled graphs, we only apply them to MMSB and LGGAN.
4. C OMPARING D IFFERENT V ARIATIONS OF LGGAN
LGGAN is a ﬂexible framework that can adopt different settings for each part of the training setup, such as the GAN framework and the model for generator and discriminator.
The GAN variations we consider are (1) LGGAN GAN, which uses the original GAN framework (Goodfellow et al., 2014), (2) LGGAN CGAN, which uses the conditional GAN framework (Mirza & Osindero, 2014), and (3)
LGGAN ACGAN, which uses the AC-GAN (Odena et al., 2017) framework.
We also compare two different discriminative models, a simple multi-layered perceptron and jumping
We evaluate these six possible setups for LGGAN on different graph-structured data to measure the quality of the learned generative models.
We run experiments on two datasets: cora small and ENZYMES.
The results are listed in Table 1.
Among all the three GAN frameworks, LGGAN ACGAN achieves the best results on both datasets regardless of which discriminative model is used.
This result matches with our expectations, since the AC-GAN framework incorporates the class information allowing it to learn a better embedding and to propagate that information to the generator.
For different discriminative models, using JK-Nets can improve the quality of generated graphs regardless of which GAN framework is used.
An interesting point is that there is a gap between
LGGAN s and LGGAN on the ENZYMES dataset that is much larger than on the citation networks.
This difference may be because the cora small dataset is composed of many small two-hop and three-hop ego networks where the structure is quite simple and uniform—so it could be easier to learn.
However, with the ENZYMES dataset, the structure is more complicated and diverse.
Therefore it reveals that the LGGAN s is unable to generalize to complex data.
In contrast, the quality of generated graphs with LGGAN using JK-Net is more consistent among different datasets, which suggests that LGGAN can adaptively adjust to different graph-structured data.
Based on these results, we use LGGAN ACGAN in the remaining experiments and refer to it as LGGAN when comparing with other baselines.
4. C OMPARING WITH O THER M ETHODS
We compare LGGAN to other methods for generating graphs—both traditional generative models such as E-R, B-A, and MMSB as well as deep generative models that were proposed recently, such as GraphRNN and DeepGMG.
DeepGMG cannot be used to generate large graphs due to its high computational complexity, so the results of DeepGMG on large graph datasets are not available.
For each method, we measure three aspects.
The ﬁrst is the quality of the generated graphs, which should be able to mimic typical topology of the training graphs.
The second is the generality, where a good generative model should be able to generalize to different and complex graph-structured data.
Then the last aspect is the scalability, where we want the model to be able to scale up to generate large networks instead of being restricted to relatively small graphs.
Table 2 lists results from our comparison.
LGGAN achieves the best performance on all datasets, with 90% decrease of MMD on average compared with traditional baselines, and 30% decrease of
MMD compared with the state-of-the-art deep learning baseline GraphRNN.
Although GraphRNN performs well on the two smaller protein-related datasets, ENZYMES and protein, it does not maintain the same performance on large datasets, such as cora and citeseer.
Scalability To evaluate the scalability of these methods, we perform experiments on two different subsets of the Cora dataset with different graph sizes: the cora small and cora datasets.
As listed in Table 2, the traditional models all create a large gap between these two datasets in terms of three evaluation metrics.
For the deep generative models, DeepGMG cannot be used to generate large
[Table]
graphs due to the computational complexity of it’s generation procedure which try to add node one by one increasingly.
And compared to GraphRNN, LGGAN MMD scores barely increase compare to the smaller dataset, suggesting that our model is more reliable and has the best ability to scale up to large graphs.
Generality To evaluate the ability of LGGAN to adapt to different graph-structured data, we evaluate the results of all methods on the different domains of citation ego-networks (Cora) and molecular protein graphs (ENZYMES).
From Table 2, LGGAN achieves more consistent results on various datasets compared to other models, where some of them suffer from the issue of generalization such as MolGAN and NetGAN which can only be used to generate speciﬁc or limited types of graphstructured data.
Some examples are visualized in Figure 3, which contains graphs generated by our model and the baselines.
Although it is not as intuitive for humans to assess as, e.g., natural images, one can still see that LGGAN appears to capture the typical structures of datasets better than other models.
4. E VALUATING L ABELED G RAPHS
To better evaluate the structure of the labeled graphs being generated, we also calculate MMD of the three graph statistics for the sub-graphs centered around each node class, taking the average MMD value across all classes.
Since among existing methods, only MMSB can be used to directly generate labels, we compare LGGAN to it using the ENZYMES datasets.
The results are listed in Table 3.
[Table]
can not only learn a good distribution of the labels, but it is also able to learn the structure within each class much more reliably than the MMSB model.
C ONCLUSION
In this work, we proposed a deep generative model using a GAN framework that generates labeled graphs.
These labeled graphs can mimic distributions of citation graphs, knowledge graphs, social networks, and more.
We also introduced an evaluation method for labeled graphs to measure how well the model learns the sub-structure of the labeled graphs.
Our model can be useful for simulation studies, especially when access to labeled graph data is limited by access or privacy concerns.
We can use these models to generate synthetic datasets or augment existing datasets, to do graphbased analyses such as communication segmentation, node classiﬁcation, anomaly detection, and link prediction.
The experiments show that it outperforms other state-of-the-art models for generating graphs while also being capable of the previously unaddressed task of generating labels for nodes.
A CKNOWLEDGEMENT
The authors would like to thank NVIDIA for donating hardware and Amazon for donating cloud computing credits to our group, we appreciate their support to our research.