Micro and Macro Level Graph Modeling for Graph Variational Auto-Encoders
Abstract
Generative models for graph data are an important research topic in machine learning.
Graph data comprise two levels that are typically analyzed separately: node-level properties such as the existence of a link between a pair of nodes, and global aggregate graph-level statistics, such as motif counts.
This paper proposes a new multi-level framework that jointly models node-level properties and graphlevel statistics, as mutually reinforcing sources of information.
We introduce a new micro-macro training objective for graph generation that combines node-level and graph-level losses.
We utilize the micro-macro objective to improve graph generation with a GraphVAE, a well-established model based on graph-level latent variables, that provides fast training and generation time for medium-sized graphs.
Our experiments show that adding micro-macro modeling to the GraphVAE model improves graph quality scores up to 2 orders of magnitude on ﬁve benchmark datasets, while maintaining the GraphVAE generation speed advantage.
Introduction: multi-level graph modeling
Many datasets contain relational information about entities and their links that can be represented as a graph.
Deep generative learning on graphs has become a popular research topic [20], with applications including molecule design [42] and recommendation [12].
It is common in graph analysis to distinguish two levels of information [10, 20]: (1) local node-level properties, such as the existence of a link between two nodes or the attribute of a node [27, 37, 19] and (2) global graph-level statistics (graph statistics for short) that depend on the entire graph, such as node degree distribution or motif counts.
Most deep Graph Generative Models (GGMs) are trained with an objective based on local properties (e.g., maximizing the probabilities of observed edges).
Figureillustrates the difference between local and global structure.
Different edges have different roles in the graph global structure.
Some edges play a critical role for the connectivity/community structure, while others are less important.
Previous GGM training is generally based on a likelihood objective that decomposes into individual edge likelihoods [47, 31, 11], and cannot capture the different role of edges.
Graph global level information on the other hand can capture the different edge roles to measure the importance of an edge in the graph structure (see also Section 4).
Whereas the sparsity of typical graphs causes difﬁculties for training objectives based on local information only [38, 27], graph statistics are typically dense.
This paper proposes a new perspective on the node/graph level dichotomy: we introduce a principled probabilistic framework that incorporates both local and global graph properties.
From the framework we derive a novel micro-macro (MM) objective function for training a GGM to match both local and global properties.
The term micro-macro originates from the philosophy of science and refers to scientiﬁc frameworks that treat local (“micro") resp. global (“macro) properties of a system as equally targets [17]. [Figure]
Advantages Micro-macro modeling increases graph realism and user control.
Compared to objective functions that are based on predicting local properties, matching graph statistics serves as a regularizer that increases the realism of the generated graph structures. This is sufﬁcient to generate realistic graphs with a fast all-at-once edge-parallel model.
(2) O’Bray et al.
[35] note that different graph statistics are important for different applications.
For example for a large payment graph recording economic transactions, a macro economist may be mainly interested in the average price level of a target goods basket.
For a central bank managing a payment system, the total number of transactions may be more important.
O’Bray et al. [35] therefore advise selecting a GGM based on the graph statistics of interest in the target application.
However, this entails searching through the space of GGMs and their hyperparameters to ﬁnd a good match with target graph statistics.
In our MM framework, the user only needs to specify the target graph statistics and learning will automatically select graph models that match them.
Approach We introduce a joint probabilistic model over both local properties and global graph statistics.
An MM objective that can be used with graph encoder-decoder models is derived as an ELBO from the joint probability model.
Our experiments focus on the GraphVAE (Graph
Variational Auto-Encoder) model where the encoder outputs a graph-level latent posterior z (graph embedding) and the decoder maps the graph embedding z to a soft adjacency matrix representing link probabilities [43].
A graph-level embedding supports modeling graph-level statistics.
We show how the recent calibrated Gaussian variational auto-encoder [41] can be adapted to model graph statistics with divergent scales.
The implementation and datasets are provided at https: //github.com/kiarashza/GraphVAE-MM .
Evaluation Evaluating GGMs has become a research topic of its own [44, 35]; see the related work section below.
Proposed metrics quantify how similar a set of generated graphs is to a set of observed graphs.
Our assessment focuses on evaluation metrics based on Graph Neural Networks (GNN-based metrics) [44].
To our knowledge, this is the most recent published evaluation method, with state-of-the-art (SOTA) performance shown in extensive experiments.
For GraphVAEs, adding micro-macro modeling improves both the realism and the diversity of the generated graphs, up to an order of magnitude on GNN-based metrics.
This is sufﬁcient to reach very competitive graph quality compared to SOTA auto-regressive benchmark models [32, 47, 11].
At the same time, one-shot graph generation with GraphVAEs is faster than sampling from auto-regressive models.
Our experiments on medium-sized graphs show that even with the overhead of micro-macro modeling, GraphVAEs maintain their generation speed advantage over auto-regressive approaches.
While our evaluation focuses on GraphVAEs, we discuss how micro-macro modeling can be applied to other GGMs to leverage graph-level statistics.
Contributions.
Our main contributions can be summarized as follows
• A new joint probabilistic model over both local graph properties and global graph-level statistics. • Deriving a joint ELBO as a new micro-macro objective function for training graph encoderdecoder models.
• An adaptation of the GraphVAE architecture [43] to learn graph embeddings using the joint micro-macro objective.
Related work
Graph Statistics as Modeling Targets.
To our knowledge, this is the ﬁrst work to develop a GGM with the objective of matching graph statistics.
Work on graph moment matching in network statistics [36, 6] has studied theoretical properties of graph moment estimators with increasing graph size.
Maximizing the likelihood of exponential random graph models [21] such as Markov Logic Networks is equivalent to maximizing entropy subject to matching a set of graph statistics [28].
Graph Statistics for Evaluating Generative Models.
The ability of generative models to match an empirical graph statistic distribution has been assessed in several research papers, which supports our approach of including them in the training objective.
Farnadi et al.
[16] compare relational models in terms of matching (expected) motif counts.
You et al.
[47] and Liao et al.
[32] use 4 graph statistics to measure the realism of generated graphs.
The question of how to evaluate a GGM has been studied in recent papers [35, 44].
An evaluation metric quantiﬁes how similar a set of generated graphs ˆ G 1 , . . .
, ˆ G m 1 is to a set of test graphs G 1 , . . .
, G m 2 .
From the graphs we can compute corresponding descriptors (e.g., statistics) ˆ x 1 , . . .
, ˆ x m 1 and x 1 , . .
, x m 2 .
The similarity of two descriptor sets can be quantiﬁed using Maximum Mean Distance (MMD).
O’Bray et al.
[35] raise several difﬁculties with this approach.
(1) Given a list of graph statistics, we obtain a list of MMD scores, rather than a single score that can be used to rank GGMs.
The statistics-based MMD scores are sensitive to hyper-parameters; different settings can lead to different model rankings even for a single graph statistic.
In perturbation studies where observed graphs are corrupted in a controlled manner, MMD scores do not correlate well with degree of perturbation.
Thompson et al.
[44] develop a recent proposal to address these issues.
They utilize a reference embedding network GNN E .
The embedder E is obtained from pretraining or from random weights and is therefore independent of any of the models under evaluation.
Given mgenerated graphs and m 2 test graphs, E provides embeddings ˆ e 1 , . . .
, ˆ e m 1 and e 1 , . . .
, e m 2 which can be compared using a distance metric such as MMD.
Extensive evaluation shows that the GGM scores provided by the reference embedding approach correlate well with perturbation degree and capture the realism and diversity of generated graphs.
To our knowledge, the GNN-based approach is the state of the art for evaluating GGMs, so we use it as our main evaluation metric.
Graph Generation Architecture.
Our novel contribution is to develop a new kind of objective function, not a new kind of GGM.
We therefore follow an AB design where we utilize an existing architecture as is and change only the training objective.
Developing GGMs is an on-going topic of research; overviews can be found in [49, 20].
Two major GGM groups are all-at-once and auto-regressive methods [20].
All-at-once generation decodes a latent variable z to generate a (soft) adjacency matrix ˜ A .
Auto-regressive methods [32, 47] generate a graph incrementally.
All-at-once methods are faster at graph generation, but tend to generate less realistic graphs.
They can use node-level representations or a graph-level representation (embedding).
We use a graph-level representation for two reasons: (1) A natural ﬁt with modeling graph-level statistics.
(2) It is known from previous work that node-level representations, while useful for many applications, do not capture enough graph structure to generate realistic graphs.
We utilize the well-known GraphVAE architecture for computing graph embeddings [43],[20, Ch.9 Sec.1.2].
For a generative model of graph statistics, we adapt the state-of-the-art calibrated Gaussian variational auto-encoder [41], which has been developed as a generative model for i.i.d.
data, but not previously been applied to graph modeling.
Data model and micro-macro objective
An attributed graph is a pair G =
( V, E ) comprising a ﬁnite set of N nodes and edges where each node is assigned an d -dimensional attribute X i .
An attributed graph can be represented by an N × N adjacency matrix A with { 0 , 1 } entries, together with an N × d node feature matrix X .
Following
Ma et al.
[33], we view the observed adjacency matrix as a sample from an underlying probabilistic adjacency matrix ˜
A with ˜ A
i,j ∈ [0 , 1] .
The sampling distribution for independent edges is given by
[Equation]
A descriptor function φ maps an adjacency matrix A to a l -dimensional graph statistic such that φ ( A ) ∈ R l [35].
We consider only descriptor functions that are permutation-equivariant [20, Ch.5].
A graph statistic represents a higher-order graph property.
Examples include a node degree histogram or motif counts; for discussion and further examples see [20, Sec.2.1.2] and Section 4 below.
3.1 Micro-macro objective
We assume a ﬁnite list of descriptor functions φ 1 , . .
, φ m that deﬁne target statistics .
For a ﬁxed attribute matrix X , a micro-macro (MM) loss is of the form
[Equation]
where A is the training graph and F u , u = 1 , . .
, m is a random variable deﬁned by applying the u -th descriptor function to the training graph.
The hyperparameter γ controls the balance between capturing micro and macro aspects of the training graph.
The micro loss L 0 decomposes into losses for each node or pair of nodes.
Well-known examples include the cross-entropy loss and max-margin loss.
In this paper we work with negative log-likelihood losses and a variational approximation based on an encoder-decoder architecture:
[Equation]
using the following notation.
• z 1 × t speciﬁes a latent t -dimensional graph embedding with prior distribution p ( z ) .
• ˜
A z is a probabilistic adjacency matrix computed as a trainable deterministic decoder function of graph embedding z .
The decoder parameters are denoted as ψ .
The edge reconstruction probability P ( A | ˜
A z ) is computed as in Equation (1).
• The conditional distribution of each graph statistic is modeled as a Gaussian N with diagonal variance parameter σ 2 u : [Equation]
The Gaussian mean is computed by applying the u -th descriptor function to the reconstructed (soft) adjacency matrix.
Even with a Gaussian conditional distribution, the marginal distribution over graph statistics, p 1 ψ , σ ( F u ) , can in principle ﬁt any distribution [26].
• | F u | is the dimensionality of target statistic F u .
Figure 2a shows a generative model diagram for Equations (2)–(4).
Normalizing each graph statistic by its dimension is important because their scales can diverge widely.
For example, transition matrices have N 2 entries, while the number of triangles is a single scalar.
The following proposition provides an ELBO for an MM loss.
Proposition 1 Let L θ ( A ) be a micro-macro loss deﬁned by Equations (2) – (4) .
Then
[Equation]
where q ϕ ( z | A , X ) =
q ( z | A , X , F 1 , . . .
, F m ) is an approximate posterior distribution with parameters ϕ .
The proof is in the Appendix section 8.1.
The basic idea is to combine ELBOs for each individual loss term.
Adding a hyperparameter β to control the relative importance of the KL-divergence (as in the β -VAE [22]), we obtain the following training objective for a set of observed training graphs A 1 , . . .
, A n :
[Equation]
In this objective [Equation] the graph statistic reconstruction loss, − γ (cid:80) mu =1 1 | F u | ln N ( F ju | φ u ( ˜ A z ) , σ 2 u I ) , acts as a regularizer with respect to the edge reconstruction loss − ln p 0 ψ ( A j | ˜
A z ) .
3.2 Implementation
We implement the micro-macro ELBO (6) utilizing graph neural networks as follows.
• The prior p ( z ) is a standard normal distribution.
• We limit the encoder input to the observed training graph, so q ( z | A , X , F 1 , . . .
, F m )
=
q ϕ ( z | A , X ) .
Therefore we can use any standard graph encoder with parameters ϕ .
The encoder deterministically maps an input graph ( A , X ) to a posterior distribution q ϕ
( z | A , X ) .
• The decoder deterministically maps a latent representation z to a probabilistic graph ˜
A z with parameters θ .
Any standard graph decoder can be used.
• For each graph-level statistic, the diagonal co-variance parameter is computed using the optimal σ -VAE method from the calibrated Gaussian framework [41].
This uses the maximum likelihood estimate given the estimated means with respect to a minibatch of size B :
[Equation]
Figure 2b plots the architecture.
The advantages of the calibrated Gaussian model for generating graph statistics are as follows.
In experiments on non-relational data, the calibrated Gaussian model achieves state-of-the-art performance.
(2) Learning a variance parameter eliminates hyperparameters compared to user-assigned weights for each graph statistic.
The variance values can be interpreted as quantifying the empirical uncertainty of a graph feature.
For example, if all training graphs exhibit a similar number of triangles, the corresponding variance parameter will be small. Table 6 in the Appendix illustrates this phenomenon in benchmark datasets.
[Figure]  Graph Statistics
The micro-macro ELBO (6) requires user-speciﬁed input graph descriptors for computing target graph statistics.
In our experiments, we utilize several types of default statistics for regularizing graph embeddings that have the following advantages.
(1) Known from prior research to be generally important for graph modeling across different domains. (2) Easy to interpret. (3) Differentiable with respect to the entries in a reconstructed soft adjacency matrix.
In an application, the default statistics can be combined with other statistics of interest.
Graph descriptors used in network analysis and graph kernels comprise two main groups [20]: (1)
Summaries of local node-level properties.
We utilize a descriptor derived from the degree histogram, and the number of triangles in a graph.(2) Higher-order proximity relations between nodes.
We utilize s -step transition probability matrices, for s = 1 , . . .
These descriptors are differentiable with respect to the entries in a reconstructed soft adjacency matrix ˜ A , which we view as representing a weighted undirected graph.
Degree Histogram.
The degree of node v i is given by d ( v i ) = (cid:80) j ˜
A ij .
We adapt the differentiable histogram layer (DHL) [45].
The DHL is based on a soft assignment of points to bins given bin centers and widths.
Binning the soft degrees forms a soft degree histogram of graph ˜
A comparable to the hard degree histogram.
In detail, the bin centers are the possible (hard) node degrees b = 0 , . . .
, N .
All widths are uniformly set to 0.1 (based on experimentation).
Then we have d ˜ A ( b ) ≡
N (cid:88) i = a ( v i , b ) , a ( v i , b ) = max { 0 , 1 − 0 . 1 · | d ( v i ) − b |} . Thus the membership a ( v i , b ) of a node in a bin b ranges from 0 to 1 and decreases with the difference between the node’s expected degree and the bin center.
The DHL assigns to each bin the sum of nodes membership in the bin.
In a naive implementation, the computational cost of ﬁnding the d ˜ A ( b ) vector is O ( N 2 ) .
In our experiments, we use N parallel processors for a near constant time to obtain the soft degree histogram. Vectors summarizing the node degree distribution have been widely used to measure graph structural similarity.
S -Step Transition Probability Kernel.
P s ( ˜ A ) is the N × N s -step transition probability matrix derived from adjacency matrix ˜ A such that [ P s ( ˜ A )] i,j is the probability of a transition from node i to node j in a random walk of s steps started from i .
The transition matrix P s ( ˜ A ) can be computed as P s ( ˜ A ) =
( D ( ˜ A ) − 1 ˜ A ) s where the entry D ( ˜ A ) ii = d ( v i ) is the degree of node i . The GraRep system learns node representations that reconstruct the s -step transition probabilities, using random walks and matrix factorization [9].
The transition probability matrix is usually dense and encodes the connectivity information of the graph.
An important difference to the adjacency matrix is that whereas adding or removing an edge changes only one adjacency, it can and often does result in a substantive change in many transition matrix elements.
The descriptor P s ( ˜ A ) thus measures the importance of an edge in the graph structure (cf.
Figure 1).
In our experiments, we compute the transition probability matrix exactly with a runtime cost of O ( N 3 ) .
Triangle Count.
The number of triangles in a simple graph A is computed by Tri ( A )
= (cid:80) i ( A 3 ) ii with a runtime cost of O ( N 3 ) .
The number of triangles is a fundamental graph statistic in network science [5] with many applications in graph mining [1].
For example it has been used to detect spamming activity and assess content quality in social networks [4], to uncover thematic structure in the world-wide web [14] and for query planning in databases [3].
It is also used extensively in exponential random graph models [21].
Empirical evaluation
We describe our baseline methods and benchmark datasets, then report comparison results.
5.1 Comparison methods
We examine the effect of micro-macro modeling on a GraphVAE architecture.
Our AB methodology is to keep the architecture the same and train the model using the joint MM ELBO (6) that combines both local and global graph properties.
We also compare the MM GraphVAE with popular GGMs.
As the architectures are not new, we describe them brieﬂy.
Our experiments used the public repositories and recommended hyperparameters by original creators; the Appendix contains further details.
GraphVAE.
A popular model that transforms a graph-level latent variable to generate a soft adjacency matrix [33, 43].
The encoder utilizes a Multi-layer Graph Convolutional Network (GCN) followed by a graph level readout and a Fully Connected Layer (FCL) that outputs ( µ, σ ) .
The decoder utilizes a network with FCLs that outputs entries of a probabilistic adjacency matrix ˜ A .
To evaluate the reconstruction probability, we use a BFS ordering of nodes [31][20, Ch.9.1.2].
GraphVAE–MM.
GraphVAE trained with the micro-macro objective 6.
Our new method.
GraphRNN.
Auto-regressive method that generates the adjacency matrix incrementally.
Each step generates one entry in the GraphRNN design, or one column in the GraphRNN-S design [47].
GRAN.
Auto-regressive method that generates a block of nodes and associated edges at a step [32].
BiGG.
Auto-regressive method that leverages graphs sparsity to avoid generating the full adjacency matrix [11].
To our knowledge BiGG achieves SOTA graph quality.
5. Benchmark datasets
Our design closely follows previous experiments on generating realistic graph structures [47, 32].
We utilize 3 synthetic, and 2 real-world datasets for the main results.
The synthetic Grid and Lobster are from previous studies [47, 32], Triangle Grid is introduced in this paper.
Protein and ogbg-molbbbp are real-world datasets from biology with information about proteins and molecules respectively.
The Appendix, section 8.2, contains further details, as well as results for 3 more real-world datasets.
Train/Test Split.
Following previous experiments [47, 30, 11] we randomly split the graphs sets into train (70%), validation (10%) and test (20%) sets.
We use the same train and test graph sets for all models.
To evaluate a trained model, we generate T new graphs to compare with the T graphs in the test set (cf.
Section 2).
Appendix Figure 5 illustrates the evaluation process.
Network modeling at the micro-level might be used for surveillance of individuals and communities.
Therefore ethical network data collection must take into account issues of consent, privacy, and bias.
None of the datasets used in this research study contain personally identiﬁable information, or offensive/harmful content about either individuals or communities.
The social impact of our work we expect to be on balance more positive than negative, because our macro-level model enhances the understanding of global network structure, not the targeting of individuals.
5.3 Evaluation
We empirically verify the effectiveness of micro-macro modeling through different evaluation metrics: Qualitative and quantitative evaluation of graph quality, generation time, and training time.
Qualitative Evaluation.
Following [47, 32] we compare the generated graphs by visual inspection.
Figure 3 provides a visual comparison of randomly selected test graphs and generated graphs.
GraphVAE-MM and BiGG achieve the best visual match.
The Appendix, section 8.5, provides more examples.
Quantitative Evaluation.
We follow the descriptor-based approach established in previous works (cf.
Section 2).
As recommended by O’Bray et al.
[35], we include scores computed from a 50/ split of the data set as an ideal score, i.e., a lower bound on the GGM scores.
(1) The GNN-based evaluation metrics [44] MMD RBF and F1 PR compare generated and test graph embeddings computed by a reference GNN with randomly initialized weights.
The reference embeddings are independent of any target graph statistics used in GraphVAE-MM.
F1 PR stands for “Improved Precision and Recall".
Precision is the percentage of generated graph embeddings that fall within the manifold of test graph embeddings, while recall is the percentage of test graph embeddings that fall within the manifold of generated graph embeddings.
This metric mainly captures the diversity of generated graphs [35].
MMD RBF compares generated graph embeddings and test graph embeddings using an RBF kernel.
This metric captures the realism of the generated graphs.
(2) A statistics-based evaluation compares generated and test graphs using evaluation statistics , namely node degree, clustering coefﬁcient, orbit counts, and the spectra of the graphs [32, 47].
We add graph diameter, a statistic commonly studied in network science [29] related to graph connectivity.
The target statistics used to train a GraphVAE-MM model are distinct from the evaluation statistics used to measure graph quality (however, node degree and Degree histogram are closely related).
For each statistic, MMD is computed using total variation (TV) distance [32]. L ob s t e r L ob s t e r
[Figure]
r i d
[Figure] Impact on GraphVAE.
Table 1 shows a very large improvement from micro-macro modeling for both MMD RBF and F1 PR.
The improvement in F1 PR ranges from 6%-40%, achieving a near perfect score.
For MMD RBF the reduction ranges from 0.06 to 0.26 which is signiﬁcant given the low ideal score.
Table 2 shows the results for evaluation graph statistics.
Micro-macro modeling reduces MMD by 1-2 orders of magnitude on almost all datasets.
In sum, MM modeling provides a large improvement in the realism and diversity of graphs generated by a GraphVAE architecture.
Lesion studies in the Appendix investigate the effects of each target statistic in isolation (Table 5).
No single statistic has the power of all three combined.
We also observe that different target statistics have different importance for different datasets; see Table 6.
[Figure]
[Table]
GraphVAE-MM vs. Benchmark GGMs.
For benchmarking we include GGMs that are known to generate realistic graphs.
Tables 3 shows the GNN-based quality scores.
Other than the most recent
BiGG method, GraphVAE-MM achieves a much better score, except for GraphRNN on the Molecule dataset.
This dataset is unusual in that it comprises only small graphs (average 23 edges).
MMD RBF scores and 3 out of 5 F1 PR scores are also better in GraphVAE-MM compared to BiGG.
Triangle
Grid shows the biggest improvement, which illustrates the usefulness of matching triangle counts for this dataset.
Appendix Table 7 shows the benchmark results of statistics-based evaluation; we summarize them here.
On synthetic graphs, the GraphVAE-MM scores are superior to or competitive with the BiGG and GRAN scores.
On the real-world graphs, the GraphVAE-MM scores are competitive with the
BiGG and GRAN scores, and superior to the other benchmarks.
Given the already strong performance of the auto-regressive models, we conclude that GraphVAE-MM generates high-quality graphs.
Generation Time.
The code for all models is run on the same system, detailed in Appendix section 8.11.
Figure 4 compares GraphVAEs to the fastest auto-regressive methods.
The auto-regressive methods require substantially more generation time than GraphVAEs. While MM modeling slows [Table]
[Table] down training, for the GraphVAE , the training time is still less than for the auto-regressive methods. The Appendix table 8 provides time measurements for all methods.
The Appendix Table 12 presents worst-case complexity bounds of the comparison methods.
We conducted further experiments on the MUTAG, PTC [46], and QM9 [43] datasets to evaluate micro-macro modeling on more real world graphs. The detailed results are in the Appendix section 8.10; we summarize them here. For all datasets, GraphVAE-MM offers much faster generation than the auto-regressive baselines.
On MUTAG and PTC, the improvements from micro-macro modeling are even better than those we reported on the Protein and ogbg-molbbbp datasets. GraphVAEMM achieves substantive improvement in generation quality over all baselines, except for BiGG on PTC, which is competitive. It was infeasible to train the auto-regressive methods on the QM datasets (except for BiGG), so we report only the comparison of GraphVAE vs. GraphVAE-MM. Micro-macro modeling brings small improvement in graph quality on QM9, which is good given
GraphVAE strong performance on QM9, compared to the other datasets.  Limitations and discussion We discuss the limitations of our GraphVAE-MM model and options for combining micro-macro modeling with other GGMs.
Strengths and Weaknesses.
GraphVAE-MM inherits the strengths of GraphVAE [20, Ch.9.1.2]: expressive power through graph embeddings, and fast generation due to all-at-once parallel edge generation.
GraphVAE–MM also inherits the known limitations of GraphVAEs: 1) We need to know a maximum number of nodes before generation.
(Smaller graphs can be generated using a mask.)
The decoder is an FCL that ouputs N × N numbers; we need to (implicitly) assume a node ordering to evaluate edge reconstruction probabilities based on the FCL output.
The dependence on a node ordering is common to both GraphVAEs and auto-regressive GGMs, and efﬁcient heuristics have been designed whose effectiveness has been conﬁrmed in experiments, including those reported in this paper.
We note that MM modeling makes GGM training less dependent on a node ordering because it also uses permutation-invariant graph statistics.
Training Time Overhead.
Evaluating the GraphVAE-MM ELBO during training incurs computational overhead compared to the GraphVAE ELBO due to computing graph statistics.
We note that evaluating the edge reconstruction probability is already expensive and can dominate training time.
Tablein the Appendix presents worst-case complexity bounds for the training time of our comparison methods, which indicats that scaling GraphVAEs to very large graphs is a challenge.
Despite the theoretical worst-case cost of evaluating graph statistics, Figure 4(right) shows that the overhead was small in our experiments with medium-size graphs, due to parallelization.
Also, approximating graph statistics is a promising avenue for signiﬁcantly reducing the training time even for large graphs signiﬁcantly [24, 15, 31, 40].
Micro-Macro Modeling for Other GGM Architectures.
An interesting hybrid architecture combines a graph-level encoder with an auto-regressive decoder [25].
This is an option for leveraging micromacro modeling in auto-regressive graph generation.
An auto-regressive decoder is much more expressive than the all-at-once standard GVAE decoders.
In a graph GAN [7, 8] the generator maps a graph latent z to an adjacency matrix, or a random walk and the discriminator classiﬁes them as real or synthetic. These models are not considered as SOTA GGMs [11].
Both random walk and adjacency matrices represent local micro-level information only [9]. A straightforward way to combine MM modeling with GANs is to augment the input to the discriminator with graph statistics computed for both real and generated graphs.
Conclusion and future work
The main idea in this paper was to model graph data jointly at two levels: a micro level based on local information (e.g. the existence of a link between two nodes) and a macro level based on aggregate graph statistics.
We described a principled joint probabilistic model for both micro and macro levels, and derived an ELBO training objective for graph encoder-decoder models.
Compared to previous micro level training objectives, the macro statistics regularize graph embeddings to match global graph statistics.
To evaluate our model, we described a set of strong default graph statistics (node degree, number of triangles, transition probabilities).
We applied the new training objective to the
GraphVAE architecture, a widely used graph generative model based on latent graph representation.
Micro-macro (MM) modeling greatly improved the quality of graphs generated by GraphVAE, to match or exceed that of benchmark models.
MM modeling maintains the speed advantages of edgeparallel all-at-once graph generation. With an efﬁcient computation of graph statistics, it provided fast training time as well compared to auto-regressive methods.
Micro-macro modeling opens a number of fruitful avenues for future work.
i)
Investigating which graph statistics are important for generating which types of graphs.
This connects with the rich area of graph kernels [34] that are often based on graph statistics.
ii)
Investigating which graph statistics are important for particular domains.
iii)
Developing a micro-macro model for other graph generative architectures, such as auto-regressive and GANs.
iv) Evaluating the impact of micro-macro modeling on other graph-level tasks, such as graph classiﬁcation.
In sum, modeling both global graph statistics and local information enhances the power of graph generation.
Compared to using local information only, graph statistics can be used to regularize graph representations to efﬁciently generate realistic graphs.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6 for details.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.2 and 8.13 for details.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sectionfor details.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 3 and Appendix for details.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code, datasets and library requirements can be found at https://github.com/ kiarashza/GraphVAE-MM (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sections 5.2 and Appendix for all experiment settings.
We also added all details, including models, generated samples, training logs and library requirements in our public repository.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] The focus of this paper is on the GNN-based metrics, tables 1 and 3, containing error bar information.
For all other compared metrics, the proposed approach typically results in 1-2 orders of magnitude of improvement.
This evidence seems to be sufﬁcient without error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 8.11 for machine details.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Reported datasets and codes are open-source and publicly available.
We cited them clearly in Tables and Context (b) Did you mention the license of the assets?
[Yes] All codes and datasets used in this study are publicly available and clearly mentioned in the Section 5.2 and 5.1.
Our implementation is publicly released under MIT license as mentioned in our repository.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We included our model implementation and it is publicly available at https: //github.com/kiarashza/GraphVAE-MM (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] As mentioned in Section 5.1, we used the original papers’ public repository; hence no consent was needed to curate this study.
Also all used datasets are open source and publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] As mentioned in Section 5.2, None of the datasets used for this research study contain any personally identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] This point is not applicable for this research study.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] This point is not applicable for this research study.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
No such participants were used for this research study. Appendix 8.1 Proof of proposition
[Equation]
A , X , F 1 , . . . , F m ) ||
p ( z ))
+ γ m (cid:88)
u
= (cid:0) [ KL ( q ϕ ( z | A , X , F 1 , . .
, F m ) || p ( z ))
(cid:1) =
E z ∼ q ϕ
( z | A , X , F 1 ,..., F m ) (cid:2) − ln p 0 ψ ( A | ˜
A z ) − γ m (cid:88) u
=
[Equation]
F
[Equation]
(10) + (1 + γm ) KL ( q ϕ ( z | A , X , F 1 , . .
, F m ) || p ( z ))
Inequality (7) adds ELBOs for each individual loss term.
Equation (8) uses the linearity of expectation.
Inequality(9) follows because and the fact that | F u | ≥ 1 .
Equation (10) collects the KL expressions into a single term, and establishes the inequality (5).
8. Benchmark datasets
Following previous studies [47, 32], we utilize three sets of synthetic and two sets of real graphs as follows.
Triangle Grid.
Includes 100 synthetic 2D graphs, regular tiling of the 2D plane with equilateral triangles, with 100 ≤ | V | < 400 [47].
Lobster.
Includes 100 synthetic graphs with 10 ≤ | V | ≤ 100 .
Generated using the code from [47].
(Square) Grid.
Includes 100 synthetic 2D graphs, regular tiling of the 2D plane with equilateral squares, with 100 ≤ | V | < 400 [47].
Protein.
Consists of 918 real-world protein graphs with 100 ≤ | V | ≤ 500 [13].
ogbg-molbbbp.
Consists of 2039 real-world molecular graphs with 2 ≤ | V | ≤ 132 [23].
8.3 Flowchart for evaluating generated graphs
Figure 5 illustrates the evaluation approach.
To split the datasets and for statistic-based evaluation metrics we use the code from [32].
For GNN-based metrics we used the code from [44].
Following
Goyal et al.
[18], for all the GGMs if a graph with disconnected components is generated, we take the maximum connected component.
8.4 Setup and hyper-parameters
GraphVAE.
We examine the effect of micro-macro modeling on the GraphVAE architecture [43].
GraphVAE encoder utilizes a two-layer GCN (256, 1026 dimension) followed by the graph-level output formulation (sum of nodes representation) ﬁnalized with FCL (1024 dimension) outputting the parameters of the variational posterior distribution.
The model assumes isotropic and standard
Gaussian distributions for the variational posterior and prior, respectively.
The GraphVAE decoder is a three-layer fully connected neural network (1024,1024,1024 dimension) that directly maps the graph representation to a probabilistic adjacency matrix.
Layers utilize Layer Normalization [2] and
LeakyReLU activation function.
Our AB methodology is to keep the architecture the same and train the model using the joint MM ELBO (6) that combines both local and global graph properties.
GraphVAE and GraphVAE-MM are trained using the Adam optimizer [26] with a learning rate of 0 .
0003 for 20 , 000 epochs except for Lobster and ogbg-molbbbp, smaller datasets, which are trained for 10 , 000 epochs.
For the hyperparameters γ and β see table 4.
Hyperparameters are selected by validation set performance.
Baselines.
For all Baselines, we used the implementation and hyperparameters setting provided by the original papers.
[Table]
8.5 Qualitative evaluation of GGMs in detail
Here we extend the visual examination of the micro-macro modeling and benchmark GGMs from
Figure 3.
The AB methodology for evaluating the micro-macro modeling is to use the same VGAE architecture, here GraphVAE section 8.4, and applying micro-macro modeling.
Figures 8, 7, 6,and 9 provide visual comparisons of benchmark GGMs and the effect of the micro-macro modeling approach.
For each model, we generate 20 samples and visually select and plot the most similar ones to the test set.
The ﬁrst block in each of the ﬁgures shows randomly selected target graphs from the test set.
The second block compares the effect of the micro-macro modeling on the GraphVAE and the last block samples graphs generated by the benchmark GGMs.
8.6 Lesion studies on MM objective components
This section drills downs into the different components of the MM objective and the importance of KL-divergence penalty.
The KL-divergence importance is studied by training the GraphVAE with β -VAE and different values for hyperparameter β which controls the relative importance of the
KL-divergence penalty.
The result shows that while the KL-divergence weighting affects the model performance, it cannot replace macro modeling.
See the ﬁrst block of table 5.
The second block of table 5 investigates the effects of each target statistic in isolation.
As shown, no single statistic has the power of all three combined.
We also observe that different statistics are more important for different datasets.
8.7 Variance parameter
Table 6 shows the values of the learned variance parameter σ 2 u by the optimal σ -VAE approach for each of the datasets.
The variance of a graph statistic can be interpreted as quantifying the empirical uncertainty of a graph statistic.
Taking for example the Triangle count statistic, the learned variance
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Table]
for the Lobster, Grid, and ogbg-molbbbp datasets is very small because there are almost no triangles in these datasets.
For the protein dataset, the learned variance is comparatively large, which indicates the number of triangles has a wider range of values in this dataset
[Table]
[Table]
8.8 Statistics-based comparison with benchmark GGMs
Table 7 shows the benchmark results of statistics-based evaluation.
On synthetic graphs, the
GraphVAE-MM scores are superior to or competitive with the BiGG and GRAN scores.
On the real-world graphs, the GraphVAE-MM scores are competitive with the BiGG and GRAN scores, and superior to the other benchmarks.
8.9 Comparison of GGMs on train and generation time
Table 8 compares benchmark GGMs and VGAEs on generation and train time for the main datasets.
The benchmark GGMs require substantially more generation time than GraphVAEs.
While MM modeling slows down training for the GraphVAE, the training time is still less than the benchmark
GGMs.
8.10 Details on Extended Experiments
This section extends the experiments on real graph sets by evaluating the micro-macro modeling on
MUTAG, PTC, and QM9 datasets.
• MUTAG is a dataset of 188 mutagenic aromatic and heteroaromatic nitro compounds [46].
• PTC is a dataset of 344 chemical compounds that reports the carcinogenicity of male and female rats [46].
• QM9 is a large dataset containing about 134k organic molecules [39].
Table 9 shows statistics-based and GNN-based evaluation of GraphVAE with and without micromacro modeling and comparison of micro-macro modeling on the train and generation.
Table
[Table]
[Table]
compares GraphVAE–MM with benchmark GGMs.
It was not feasible to train all auto-regressive benchmarks on QM9, so we report results only for the GraphVAE architecture with and without micro-macro modeling.
GraphVAE and GraphVAE-MM are trained for 20 , 000 epochs except for
QM9, a large dataset with more than 134 k graphs, which are trained for 100 epochs.
The experiments follow the setting in the main body of the paper, for details see section 8.4.
Impact on GraphVAE.
Table 9 shows a very large improvement from micro-macro modeling for
MMD RBF of generated graphs, especially for PTC the MMD RBF is reduced by more than one order of magnitude.
The diversity of generated graphs, F1 PR, also substantially increased, The magnitude of the increase is up to 53%.
For QM9, also the reality of generated graphs slightly improved.
GraphVAE already had a strong performance on QM9, compared to the other datasets.
To calculate the ideal score, we used two randomly selected subsets of QM9 dataset, each including 5 k graphs, because statistic-based evaluation metrics are slow to compute [44].
GraphVAE-MM vs. Benchmark GGMs.
As table 10a shows, GraphVAE-MM beats the baselines on the MMD RBF, and is very competitive on the F1 PR, though worse than BiGG.
In addition, in our experiments, GraphVAE-MM is much faster than the auto-regressive baselines in generation time, see 10c.
8.11 System architecture
The code for all models is run on the same system, an Intel(R) Core(TM) i9-9820X CPU 3.30GHz and
Nvidia TITAN RTX GPU with TU102-core.
Because of package compatibility issues, GraphRNN(-S) is run on an Intel(R) Core(TM) i7-5820K CPU 3.30GHz and a GM200 GeForce GTX TITAN X.
[Table]
[Table]
8.12 Complexity bounds
Table 11 gives complexity bounds for the different default statistics we use in this paper.
Tablepresents computational complexity of GGMs.
The bounds are based on the literature and the analysis presented in this paper.
8.13 Societal impact
Graph generation could have both positive and negative Societal impact, depending on the application domain.
On the positive side, graphs can represent molecules, and graph modeling supports medical discovery.
On the negative side, network analysis ,as a ﬁeld, has potential to increase and misuse control over network participants.
For example, to motivate surveillance violations of privacy in targeting recommendations, or identify users through their social links.
However, these harms can be mitigated by strengthening privacy protections during data collection.
Furthermore, network analysis can provide signiﬁcant societal beneﬁt, for example, by highlighting the existence and situation of marginalized communities and understanding the ﬂow of inﬂuence and (mis)information in social networks.
The main contribution of our work supports more beneﬁcial uses by enhancing the understanding of global network structure, as opposed to surveillance and the potentially harmful targeting of individuals.
8.14 Code overview
The implementation is provided at https://github.com/kiarashza/GraphVAE-MM .
main.py includes the training pipeline and also micro-macro objective functions implementation.
Source codes for loading real graph datasets and generating synthetic graphs are included in data.py.
All the Python packages used in our experiments are provided in environment.yml.
Generated graph samples for each of the datasets are provided in the "ReportedResult/" directory, both in the pickle and png format.
This directory also includes the log ﬁles and hyperparameters details used to train the GraphVAE
[Table]
[Table]
[Table]
[Table]
[Table]