R ECOVERING B ARAB ´ ASI -A LBERT P ARAMETERS OF G RAPHS THROUGH D ISENTANGLEMENT
A BSTRACT
Classical graph modeling approaches such as Erd˝os-R´enyi (ER) random graphs or Barab´asi-Albert (BA) graphs, here referred to as stylized models, aim to reproduce properties of real-world graphs in an interpretable way.
While useful, graph generation with stylized models requires domain knowledge and iterative trial and error simulation.
Previous work by Stoehr et al.
(2019) addresses these issues by learning the generation process from graph data, using a disentanglement-focused deep autoencoding framework, more speciﬁcally, a β -Variational Autoencoder ( βVAE).
While they successfully recover the generative parameters of ER graphs through the model’s latent variables, their model performs badly on sequentially generated graphs such as BA graphs, due to their oversimpliﬁed decoder.
We focus on recovering the generative parameters of BA graphs by replacing their β -VAE decoder with a sequential one.
We ﬁrst learn the generative BA parameters in a supervised fashion using a Graph Neural Network (GNN) and a Random Forest Regressor, by minimizing the squared loss between the true generative parameters and the latent variables.
Next, we train a β -VAE model, combining the GNN encoder from the ﬁrst stage with an LSTM-based decoder with a customized loss.
I NTRODUCTION
Knowing the generative procedure of a graph and its parameters allows for predicting, controlling and understanding its driving principles (Brugere et al., 2018; Barab ´ asi & P ´ osfai, 2016; Newman, 2010).
One way to uncover the underlying generative parameters of such a procedure is through disentanglement – an approach which has recently gathered a large interest in the research community.
The aim is to ﬁnd a representation of the data where the variables are disentangled in the sense that each variable corresponds to at most one generative parameter.
Disentangled representations have been successfully found for image data (Burgess et al., 2019; van Steenkiste et al., 2019; Leeb et al., 2020;
Besserve et al., 2019).
However, disentanglement of graph data is largely unexplored, with a few exceptions (Ma et al., 2019; Guo et al., 2020;
Stoehr et al., 2019).
Previous work (Stoehr et al., 2019) has proposed a disentanglement framework with which they recover the generative parameters of graphs generated by stylized graph models.
Stylized graph generation models are graph generation algorithms that typically follow a ﬁxed procedure in order to capture certain characteristics of real-world graphs.
While useful, these models require pre-speciﬁed parameters.
Furthermore, ﬁnding the appropriate values of these parameters typically demands either domain knowledge or time-consuming simulations.
With a disentanglement framework however, one can ﬁnd these generative parameters unsupervised from a given dataset.
A commonly used stylized model is the Erd˝os-R´enyi (ER) random graph generator, which deﬁnes the number of nodes n and the uniform linking probability p between any two nodes in the graph.
In this generation process the individual edges are sampled independently of each other and the graph can thus be generated accurately ”one-shot”, meaning that the procedure can be non-iterative.
However, most real-world graph datasets are generated through a sequential process.
Therefore, the ER model does not capture many of the properties typically observed in real-world graphs, such as the tendency of nodes to link to more connected nodes, a phenomenon known as preferential attachment.
Graphs that exhibit this property are called scale-free, and they evolve dynamically, continuously growing with the addition of new nodes.
The Barab´asi-Albert (BA)
Preferential Attachment model (Barab´asi & Albert, 1999) is a stylized model that was devised in order to more accurately model such graphs, in which edges are generated sequentially.
As the properties of scale-free graphs are inherently the result of an iterative process, we are not aware of a method for generating them ”one-shot”.
The β -Variational Autoencoder ( β -VAE) architecture (Higgins et al., 2017) suggested by Stoehr et al.
(2019) is capable of both generating graphs and recovering compact, disentangled representations of their generative parameters.
While their approach performs well when applied to ER graphs, their decoder generates the graphs one-shot which hinders their performance on the scale-free BA graphs.
Focusing on data generated from the non-linear extension of BA (Onody & de Castro, 2004), the main contribution of this work is creating a graph VAE with a sequential decoder, for ﬁnding compact graph representations corresponding to the generative parameters of BA graphs.
In this paper we ﬁrst show that it is possible to recover the generative parameters, by learning them in a supervised fashion using a Graph Neural Network (GNN) and a Random Forest (RF) model, using the mean squared error (MSE) loss between the true generative parameters and the predictions.
Next, we train a β -VAE model, combining the GNN encoder from the previous stage with an LSTM-based decoder, in order to obtain a disentangled representation of the generative parameters.
Creating a sequential graph decoder network is especially challenging, as it should mimic a sequential random generation process, with random sampling at each step.
Such a process is not naturally differentiable with respect to the latent variables, and typically requires tricks to enable backpropagation.
As such, we adapt the LSTM model to the graph setting for this task.
M ODELS AND M ETHODS
We use a synthetic dataset that is generated with the BA model.
The modeling approach is split into two stages: parameter prediction via supervised training and disentanglement via unsupervised training.
In the ﬁrst part we predict the generative factors of the BA graphs using both a Random
Forest and a GNN.
In the second part, we implement a β -VAE with a sequential LSTM decoder network, to ﬁnd disentangled latent representations of the generative parameters of the BA graphs.
BA Graph Generation The non-linear extension of the BA model is parameterized by the number of nodes n , an edge generation integer m and the exponential of the node degree α .
In the standard
BA model (Barab ´ asi & Albert, 1999), we set α = 1 and get a linear dependency on the node degrees.
In real-world scale-free graphs, preferential attachment can also be non-linearly dependent on the node degrees and the extended BA model (Onody & de Castro, 2004) accounts for these scenarios through the parameter α .
The description of the generative process can be found in Appendix A. and illustrations of BA graphs with different values of α are shown in Figure 2.
2. P REDICTING G ENERATIVE P ARAMETERS
In this stage, we learn the generative parameters in a supervised manner.
We train a model to output predictions for the generative parameters, and take the MSE between the model’s output and the true generative parameters to be our training loss.
As a ﬁrst step, we want to verify that the generative parameters are learnable from the data.
To do so, we extract feature vectors from the input graphs, and then train a Random Forest (RF) regressor to predict the parameters from these vectors.
Note that this step is only for the purpose of veriﬁcation, as the RF model does not allow for backpropagation and can thus not be included in an autoencoder framework.
Furthermore, a non-neural model requires manual feature extraction, whereas a GNN can automatically extract features from the graphs.
As such, we proceed to implement a GNN to predict the generative parameters, being adapted from Stoehr et al.
The outputs of the GNN can be interpreted as the latent variables in an autoencoder model, with a BA generator as the decoder.
The BA generator is non-differentiable and lacks learnable parameters however, which prevents unsupervised training via minimizing the reconstruction loss.
A high-level architecture can be seen in the top part of Figure 1.
2. G ENERATIVE P ARAMETERS AS D ISENTANGLED L ATENT V ARIABLES
To obtain a VAE, we keep the GNN encoder used in the prediction stage, and implement a differentiable decoder in the form of an LSTM.
The latent variables produced by the encoder are passed through a deconvolutional layer to obtain N graph representations, N being the number of generations steps.
These are then passed to the LSTM decoder.
The model is trained with the standard β -VAE objective, composed of a scaled KL-divergence term enforcing the variational posterior to be similar to an isotropic Gaussian, and a reconstruction term comparing the input and output graphs.
We use a customized reconstruction term to incorporate a set of constraints to adhere to a graph generation process; namely enforcing symmetry, no self-connections, increasing number of nodes and edges in each step, and starting with an empty graph.
This is added in order to replicate the generation process of a sequential stylized model.
A high-level architecture of this stage can be seen in the bottom part of Figure 1.
Mutual Information Gap Among the many metrics for assessing the quality of disentangled latent representations 1 , we choose MIG as it captures axis-alignment and is unbiased (Chen et al., 2018).
The MIG of latent variables z 1 ,...,J with respect to generative factors v 1 ,...,K is deﬁned as:
[Equation]
where I is the mutual information, H is the entropy and j ( k ) =
arg max j I ( z j ; v k ) .
Intuitively, the
MIG averages the difference between the largest mutual information and the second largest mutual information for each generative parameter with respect to the latent variables.
It is then normalized by the entropy to be between 0 and 1 , where a larger MIG value represents better disentanglement.
R ESULTS
We assess the performance of the supervised GNN model in predicting the generative parameters, for both linear and non-linear BA graphs.
The training and test sets are of size 16 , 000 and 4 ,respectively.
The number of nodes n is ﬁxed to be 50 , m is sampled as an integer uniformly at random between 1 and 49 , and for the non-linear case α is sampled uniformly at random between 1 / 3 and 3 in order to capture graphs of both sub-linear and super-linear degree distribution 2 .
The results of the two learning tasks are shown in Table 1 together with the results from the RF validation model.
Further implementation details can be found at
[Table] [Table] The GNN learns the parameters of the linear BA model with a satisfactory performance, but does not manage to learn α , while the RF model manages to learn α with a signiﬁcantly lower MSE and a Pearson correlation coefﬁcient of 0.89 between the true and predicted α , indicating that α is in fact learnable from the graph data.
The predicted parameter values can be used to generate new graphs, using the BA graph generator.
These can be compared to the original graphs for performance assessment, see Appendix A.2 for more details.
For the LSTM-based β -VAE, we compute the MIG score between the generative parameters and the latent variables.
The model is trained on two BA datasets, a linear one with α = 1 , and a non-linear one with α ∈ ( 1 3 , 3) .
Both datasets consist of 200 graphs, with node sizes varying from 3 to 12 , and the model is trained for 200 epochs.
The MIG scores are presented in Table 2.
We see that when trained on linear BA data the model displays the best performance, as expected since this is an easier task.
We further note that the parameters n and m in the BA model are not independent.
Since independence is enforced by the KL-term when training a β -VAE we also examine the MIG with respect to the parameter mn instead of m .
This parameter is independent from n , but together with n completely describes m .
We observe in Table 2 that the MIG scores for mn is indeed higher.
D ISCUSSION
Encoder Architectures In the prediction setting, we achieve satisfactory results for learning the parameters with a simple Random Forest model that uses manually extracted features as input.
We do not manage to learn α with the GNN, which also contributes to poor results in the disentanglement stage, where we use this architecture as the encoder.
However, the performance of the RF model attests that the graph data holds sufﬁcient information about the α parameter for a machine learning model to learn it.
We thus believe that other GNN architectures should be experimented with to obtain better performance on this task.
In particular, the graph convolutional layer used in our GNN requires adjacency matrices as input and output and as such depends on a ﬁxed number of nodes of the graph 3 .
There exist various graph convolutional layers that are invariant to the number of nodes, outputting embeddings on node level instead of graph level (Bacciu et al., 2020), that could be used in future work.
Categorical Latent Variables We use Gaussian distributions as priors on the β -VAE’s latent variables.
However, both n and m are categorical variables, and α is bounded to a ﬁxed-length interval, so Gaussians do not provide an optimal ﬁt.
In the linear BA case, we could sample from a discrete distribution, for instance using the Gumbel-Softmax trick (Jang et al., 2017).
Implementing this in the non-linear case is not as straightforward, since α is not discrete.
Generation Model A major challenge is the backpropagation through the decoder model, which essentially requires a graph generation process that both sequentially generates a graph and is differentiable with respect to the latent variables.
The LSTM model can be considered sequential as it takes a graph representation as input in every time step.
However, it conditions on the full generation history which is expensive and not necessary in order to replicate a BA graph generation process, in which each generation step conditions only on the previous state of the graph. Thus, a model with a ﬁrst-order Markov assumption would have been sufﬁcient. In real-world graphs however, the generation process might depend on earlier states of the graph as well, motivating our choice of a model with long-term dependency. We leave it for future work to discover the dynamics and dependencies of real-world graph generation.
Other drawbacks of the LSTM model are long training times as well as it not being a natural ﬁt for graph data, requiring tricks to output an adjacency matrix.
We note the existence of sequential GNN architectures (Li et al., 2018; Liu et al., 2018).
However, the former conditions on the full generation history and the latter is tailored speciﬁcally towards molecule generation.
Node Attributes In this work, we focus on graphs without node attributes.
However, the same disentanglement methods can also be applied to graphs where the nodes come with attributes – a feature present in most real-world graphs.
An interesting future direction could be to identify and disentangle the dependency of node attributes on the graph topology.
C ONCLUSION
We recovered the generative parameters of the Barab´asi-Albert model and its non-linear extension through disentanglement.
To this end, we emphasized the differences between sequential and nonsequential graph generators, and the scarcity of interpretable neural graph generators adhering to a sequential generation process.
We hope to point out a promising and fascinating research direction with plenty of room for future improvement
A.
B ARAB ´ ASI -A LBERT GENERATION PROCEDURE
The non-linear extension of BA (Onody & de Castro, 2004) is parameterized by the number of nodes n , an edge generation integer m and the exponential of the node degree α .
The generative process is deﬁned as follows:
Initialize a graph with m nodes.
At each step, until graph has n nodes, add a new node and connect it to m nodes randomly sampled from the following categorical distribution:
[Equation]
where k i is the degree of node i , and p ( k i ) is the probability that the new node connects to node i .
A. P ERFORMANCE A SSESSMENT OF P ARAMETER P REDICTION
For each test graph, we compute the standard deviation of degree and betweenness centrality for 1 , 000 generated graphs and compare to those of the input graph.
We consider the model’s predictions of parameters reasonable if the input graph properties fall within the distribution given by the generated graphs.
The plots for a sample of three test graphs from the linear & non-linear BA models are shown in Figure 3 & 4 respectively.
As seen in these examples, the measures of graphs generated from the linear BA model more closely follow their induced graphs’ distribution.