C ONTINUOUS G RAPH F LOW
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose Continuous Graph Flow , a generative continuous ﬂow based method that aims to model complex distributions of graph-structured data.
Once learned, the model can be applied to an arbitrary graph, deﬁning a probability density over the random variables represented by the graph.
It is formulated as an ordinary differential equation system with shared and reusable functions that operate over the graphs.
This leads to a new type of neural graph message passing scheme that performs continuous message passing over time.
This class of models offers several advantages: a ﬂexible representation that can generalize to variable data dimensions; ability to model dependencies in complex data distributions; reversible and memory-efﬁcient; and exact and efﬁcient computation of the likelihood of the data.
We demonstrate the effectiveness of our model on a diverse set of generation tasks across different domains: graph generation, image puzzle generation, and layout generation from scene graphs.
Our proposed model achieves signiﬁcantly better performance compared to state-of-the-art models.
I NTRODUCTION
Modeling and generating graph-structured data has important applications in various scientiﬁc ﬁelds such as building knowledge graphs (Lin et al., 2015; Bordes et al., 2011), inventing new molecular structures (Gilmer et al., 2017) and generating diverse images from scene graphs (Johnson et al., 2018).
Being able to train expressive graph generative models is an integral part of AI research.
Signiﬁcant research effort has been devoted in this direction.
Traditional graph generative methods (Erd ˝ os & R ´ enyi, 1959; Leskovec et al., 2010; Albert & Barab ´ asi, 2002; Airoldi et al., 2008) are based on rigid structural assumptions and lack the capability to learn from observed data.
Modern deep learning frameworks within the variational autoencoder (VAE) (Kingma & Welling, 2014) formalism offer promise of learning distributions from data.
Speciﬁally, for structured data, research efforts have focused on bestowing VAE based generative models with the ability to learn structured latent space models (Lin et al., 2018;
He et al., 2018; Kipf & Welling, 2016).
Nevertheless, their capacity is still limited mainly because of the assumptions placed on the form of distributions.
Another class of graph generative models are based on autoregressive methods (You et al., 2018; Kipf et al., 2018).
These models construct graph nodes sequentially wherein each iteration involves generation of edges connecting a generated node in that iteration with the previously generated set of nodes.
Such autoregressive models have been proven to be the most successful so far.
However, due to the sequential nature of the generation process, the generation suffers from the inability to maintain long-term dependencies in larger graphs.
Therefore, existing methods for graph generation are yet to realize the full potential of their generative power, particularly, the ability to model complex distributions with the ﬂexibility to address variable data dimensions.
Alternatively, for modeling the relational structure in data, graph neural networks (GNNs) or message passing neural networks (MPNNs)
(Scarselli et al., 2009; Gilmer et al., 2017; Duvenaud et al., 2015; Li et al., 2017; Kipf & Welling, 2017; Santoro et al., 2017; Zhang et al., 2018) have been shown to be effective in learning generalizable representations over variable input data dimensions.
These models operate on the underlying principle of iterative neural message passing wherein the node representations are updated iteratively for a ﬁxed number of steps.
Hereafter, we use the term message passing to refer to this neural message passing in GNNs.
We leverage this representational ability towards graph generation.
In this paper, we introduce a new class of models – Continuous Graph Flow (CGF): a graph generative model based on continuous normalizing ﬂows (Chen et al., 2018; Grathwohl et al., 2019)
that
[Figure]
generalizes the message passing mechanism in GNNs to continuous time.
Speciﬁcally, to model continuous time dynamics of the graph variables, we adopt a neural ordinary different equation (ODE) formulation.
Our CGF model has both the ﬂexibility to handle variable data dimensions (by using GNNs) and the ability to model arbitrarily complex data distributions due to free-form model architectures enabled by the neural ODE formulation.
Inherently, the ODE formulation also imbues the model with following properties: reversibility and exact likelihood computation.
Concurrent work on Graph Normalizing Flows (GNF) (Liu et al., 2019) also proposes a reversible graph neural network using normalizing ﬂows.
However, their model requires a ﬁxed number of transformations.
In contrast, while our proposed CGF is also reversible and memory efﬁcient, the underlying ﬂow model relies on continuous message passing scheme.
Moreover, the message passing in GNF involves partitioning of data dimensions into two halves and employs coupling layers to couple them back.
This leads to several constraints on function forms and model architectures that have a signiﬁcant impact on performance (Kingma & Dhariwal, 2018).
In contrast, our CGF model has unconstrained (free-form) Jacobians, enabling it to learn more expressive transformations.
We demonstrate the effectiveness of our CGF-based models on three diverse tasks: graph generation, image puzzle generation, and layout generation based on scene graphs.
Experimental results show that our proposed model achieves signiﬁcantly better performance than state-of-the-art models.
P RELIMINARIES
Graph neural networks.
Relational networks such as Graph Neural Networks (GNNs) facilitate learning of non-linear interactions using neural networks.
In every layer (cid:96) ∈ N of a GNN, the embedding h ( (cid:96) ) i corresponding to a graph node accumulates information from its neighbors of the previous layer recursively as described below.
[Equation]
where the function g is an aggregator function, S ( i ) is the set of neighbour nodes of node i , and f ij is the message function from node j to node i , x i ( (cid:96) − 1) and x j ( (cid:96) − 1) represent the node features corresponding to node i and j at layer ( (cid:96) − 1) respectively.
Our model uses a restricted form of
GNNs where embeddings of the graph nodes are updated in-place ( x i ← h i ), thus, we denote graph node as x and ignore h hereafter.
These in-place updates allow using x i in the ﬂow-based models while maintaining the same dimensionality across subsequent transformations.
Normalizing ﬂows and change of variables.
Flow-based models enable construction of complex distributions from simple distributions (e.g. Gaussian) through a sequence of invertible mappings (Rezende & Mohamed, 2015).
For instance, a random variable z is transformed from an initial state density z 0 to the ﬁnal state z K using a chain of K invertible functions f k described as:
[Equation]
The computation of log-likelihood of a random variable uses change of variables rule formulated as:
[Equation]
where ∂f k ( z k − 1 ) /∂
z k is the Jacobian of f k for k ∈ { 1 , 2 , ..., K } .
Continuous normalizing ﬂows.
Continuous normalizing ﬂows (CNFs) (Chen et al., 2018; Grathwohl et al., 2019) model the continuous-time dynamics by pushing the limit on number of transformations.
Given a random variable z , the following ordinary differential equation (ODE) deﬁnes the change in the state of the variable.
[Equation]
Chen et al.
(2018) extended the change of variables rule described in Eq.
3 to continuous version.
The dynamics of the log-likelihood of a random variable is then deﬁned as the following ODE.
[Equation]
Following the above equation, the log likelihood of the variable z at time t 1 starting from time t 0 is
[Equation]
where the trace computation is more computationally efﬁcient than computation of the Jacobian in Equation (4).
Building on CNFs, we present continuous graph ﬂow which effectively models continuous time dynamics over graph-structured data.
C ONTINUOUS GRAPH FLOW
Given a set of random variables X containing n related variables, the goal is to learn the joint distribution p ( X ) of the set of variables X .
Each element of set X is x i ∈
R m where i = 1 , 2 . . . , n and m represents the number of dimensions of the variable.
For continuous time dynamics of the set of variables X , we formulate an ordinary differential equation system as follows:
[Equation]
where ˙ x i =
d x i /dt and X ( t ) is the set of variables at time t .
The random variable x i at time tfollows a base distribution that can have simple forms, e.g. Gaussian distributions.
The function f
i implicitly deﬁnes the interaction among the variables.
Following this formulation, the transformation of the individual graph variable is deﬁned as
[Equation]
This provides transformation of the value of the variable x
i from time t 0 to time t 1 .
3. C ONTINUOUS MESSAGE PASSING
The form in Eq. 8 represents a generic multi-variate update where interaction functions are deﬁned over all the variables in the set X .
However, the functions do not take into account the relational structure between the graph variables.
To address this, we deﬁne a neural message passing process that operates over a graph by deﬁning the update functions over variables according to the graph structure.
This process begins from time t 0 where each variable x
i ( t 0 ) only contain local information.
At time t , these variables are updated based on the information gathered from other neighboring variables.
For such updates, the function f i in Eq. 8 is deﬁned as:
[Equation]
(9) where ˆ f ij ( · ) is a reusable message function and used for passing information between variables x i and x j , S ( i ) is the set of neighboring variables that interacts with variable x
i , and g ( · ) is a function that aggregates the information passed to a variable.
The above formulation describes the case of pairwise message functions, though this can be generalized to higher-order interactions.
We formulate it as a continuous process which eliminates the requirement of having a predetermined number of steps of message passing.
By further pushing the message passing process to update at inﬁnitesimally smaller steps and continuing the updates for an arbitrarily large number of steps, the update associated with each variable can be represented using shared and reusable functions as the following ordinary differential equation (ODE) system.
[Equation]
where ˙ x i =
d x i /dt .
Performing message passing to derive ﬁnal states is equivalent to solving an initial value problem for an ODE system.
Following the ODE formulation, the ﬁnal states of variables can be computed as follows.
This formulation can be solved with an ODE solver.
[Equation]
3. C ONTINUOUS MESSAGE PASSING FOR DENSITY TRANSFORMATION
Continuous graph ﬂow leverages the continuous message passing mechanism (described in Sec. 3.1) and formulates the message passing as implicit density transformations of the variables (illustrated in Figure 1).
Given a set of variables X with dependencies among them, the goal is to learn a model that captures the distribution from which the data were sampled.
Assume the joint distribution p ( X ) at time t 0 has a simple form such as independent Gaussian distribution for each variable x
i ( t 0 ) .
The continuous message passing process allows the transformation of the set of variables from X ( t 0 ) to X ( t 1 ) .
Moreover, this process also converts the distributions over variables from simple base distributions to complex data distributions.
Building on the independent variable continuous time dynamics described in Eq.
5, we deﬁne the dynamics corresponding to related graph variables as:
[Equation]
where F represents a set of reusable functions incorporating aggregated messages.
Therefore, the joint distribution of set of variables X can be described as:
[Equation]
Here we use two types of density transformations for message passing: (1) generic message transformations – transformations with generic update functions where trace in Eq.
13 can be approximated instead of computing by brute force method, and (2) multi-scale message transformations – transformations with generic update functions at multiple scales of information.
Generic message transformations.
The trace of Jacobian matrix in Eq.
13 is modeled using a generic neural network function.
The likelihood is deﬁned as:
[Equation]
where F denotes a neural network for message functions, and (cid:15) is a noise vector and usually can be sampled from standard Gaussian or Rademacher distributions.
Multi-scale message transformations.
As a generalization of generic message transformations, we design a model with multi-scale message passing to encode different levels of information in the variables.
Similar to Dinh et al.
, we construct our multi-scale CGF model by stacking several blocks wherein each ﬂow block performs message passing based on generic message transformations.
After passing the input through a block, we factor out a portion of the output and feed it as input to the subsequent block.
The likelihood is deﬁned as:
[Equation]
where b = 1 , 2 , . . .
, ( B − 1) with B as the total number of blocks in the design of the multi-scale architecture.
Assume at time t b ( t 0 <
t b < t 1 ), X ( t b ) is factored out into two.
We use one of these (denoted as ˜ X ( t b ) ) as the input to the ( b + 1) th block.
Let ˜ X
( t b ) be the input to the next block, the density transformation is formulated as:
[Equation]
E XPERIMENTS
To demonstrate the generality and effectiveness of our Continuous Graph Flow (CGF), we evaluate our model on three diverse tasks: (1) graph generation, (2) image puzzle generation, and (3) layout generation based on scene graphs.
Graph generation requires the model to learn complex distributions over the graph structure.
Image puzzle generation requires the model to learn local and global correlations in the puzzle pieces.
Layout generation has a diverse set of possible nodes and edges.
These tasks have high complexity in the distributions of graph variables and diverse potential function types.
Together these tasks pose a challenging evaluation for our proposed method.
4. G RAPH G ENERATION
Datasets and Baselines.
We evaluate our model on graph generation on two benchmark datasets
EGO - SMALL and COMMUNITY - SMALL
(You et al., 2018) against four strong state-of-the-art baselines: VAE-based method (Simonovsky & Komodakis, 2018), autoregressive graph generative model GraphRNN
(You et al., 2018) and DeepGMG (Li et al., 2018), and Graph normalizing ﬂows (Liu et al., 2019).
Evaluation.
We conduct a quantitative evaluation of the generated graphs using Maximum Mean
Discrepancy (MMD) measures proposed in G RAPH RNN (You et al., 2018).
The MMD evaluation in
G RAPH RNN was performed using a test set of N ground truth graphs, computing their distribution over the nodes, and then searching for a set of N generated graphs from a larger set of samples generated from the model that best matches this distribution.
As mentioned by Liu et al.
(2019), this evaluation process would likely have high variance as the graphs are very small.
Therefore, we also performed an evaluation by generating 1024 graphs for each model and computing the MMD distance between this generated set of graphs and the ground truth test set.
Baseline results are from
Liu et al.
Implementation details refer to Appendix A.
Results and Analysis.
Table 1 shows the results in terms of MMD.
Our CGF outperforms the baselines by a signiﬁcant margin and also the concurrent work GNF.
We believe our CGF outperforms
GNF because it employs free-ﬂow functions forms unlike GNF that has some contraints necessitated by the coupling layers.
Fig.
2 visualizes the graphs generated by our model.
Our model can capture the characteristics of datasets and generate diverse graphs that are not seen during the training.
For additional visualizations and comparisons, refer to the Appendix A.
4. I MAGE PUZZLE GENERATION
Task description.
We design image puzzles for image datasets
to test model’s ability on ﬁtting very complex node distributions in graphs.
Given an image of size W × W , we design a puzzle by dividing the original image into non-overlapping unique patches.
A puzzle patch is of size w × w , in which w represents the width of the puzzle.
Each image is divided into p = W/w puzzle patches both horizontally and vertically, and therefore we obtain P =
p × p patches in total.
Each patch corresponds to a node in the graph.
To evaluate the performance of our model on dynamic graph sizes, instead of training the model with all nodes, we sample ˜ p
adjacent patches where ˜ p is uniformly sampled from { 1 , . . .
, P } as input to the model during training and test.
In our experiments, we use
[Table]
Datasets and baselines.
We design the image puzzle generation task for three datasets: MNIST (LeCun et al., 1998), CIFAR10 (Krizhevsky et al., 2009), and CelebA (Liu et al., 2015).
CelebA dataset does not have a validation set, thus, we split the original dataset into a training set of 27,000 images and test set of 3,000 images as in (Kingma & Dhariwal, 2018).
We compare our model with six state-of-the-art VAE based models: (1) StructuredVAE (He et al., 2018), (2) Graphite (Grover et al., 2019), (3) Variational message passing using structured inference networks (VMP-SIN)
(Lin et al., 2018), (4) BiLSTM + VAE: a bidirectional LSTM used to model the interaction between node latent variables (obtained after serializing the graph) in an autoregressive manner similar to Gregor et al.
(2015), (5) Variational graph autoencoder (GAE) (Kipf & Welling, 2016), and (6) Neural relational inference (NRI) (Kipf et al., 2018): we adapt this to model data for single time instance and model interactions between the nodes.
Results and analysis.
We report the negative log likelihood (NLL) in bits/dimension (lower is better).
The results in Table 2 indicate that CGF signiﬁcantly outperforms the baselines.
In addition to the quantitative results, we also conduct sampling based evaluation and perform two types of generation experiments: (1) Unconditional Generation:
Given a puzzle size p , p 2 puzzle patches are generated using a vector z sampled from Gaussian distribution (refer Fig. 3(a)); and (2) Conditional
Generation:
Given p 1 patches from an image puzzle having p 2 patches, we generate the remaining ( p 2 − p 1 ) patches of the puzzle using our model (see Fig. 3(b)).
We believe the task of conditional generation is easier than unconditional generation as there is more relevant information in the input during ﬂow based transformations.
For unconditional generation, samples from a base distribution (e.g. Gaussian) are transformed into learnt data distribution using the CGF model.
For conditional generation, we map x a ∈ X a where X a ⊂ X to the points in base distribution to obtain z a and
[Figure]
[Table]
Grover et al., 2019)
[Table] [Table]
Kipf & Welling, 2016)
[Table]
L AYOUT GENERATION FROM SCENE GRAPHS
Task description and evaluation metrics.
Layout generation from scene graphs is a crucial task in computer vision and bridges the gap between the symbolic graph-based scene description and the object layouts in the scene (Johnson et al., 2018;
Zhao et al., 2019; Jyothi et al., 2019).
Scene graphs represent scenes as directed graphs, where nodes are objects and edges give relationships between objects.
Object layouts are described by the set of corresponding bounding box annotations (Johnson et al., 2018).
Our model uses scene graph as inputs (nodes correspond to objects and edges represent relations).
An edge function is deﬁned for each relationship type.
The output contains a set of object bounding boxes described by { [ x i , y i , h i , w i ] } ni =1 , where x i , y i are the top-left coordinates, and w i ,
h i are the bounding box width and height respectively.
We use negative log likelihood per node (lower is better) for evaluating models on scene layout generation.
Datasets and baselines.
Two large-scale challenging datasets are used to evaluate the proposed model: Visual Genome (Krishna et al., 2017) and COCO-Stuff (Caesar et al., 2018) datasets.
Visual Genome contains 175 object and 45 relation types.
The training, validation and test set contain 62565, 5506 and 5088 images respectively.
COCO-Stuff dataset contains 24972 train, 1024 validation, and 2048 test scene graphs.
We use the same baselines as in Sec.
Results and analysis.
We show quantitative results in Table 3 against several state-of-the-art baselines.
Our CGF model signiﬁcantly outperforms these baselines in terms of negative log likelihood.
Moreover, we show some qualitative results in Fig.
Our model can learn the correct relations deﬁned in scene graphs for both conditional and unconditional generation, Furthermore, our model is capable to learn one-to-many mappings and generate diverse of layouts for the same scene graph.
[Figure]
[Table]
4. A NALYSIS : G ENERALIZATION T EST
To test the generalizability of our model to variable graph sizes
[Table]
we design three different evaluation settings and test it on image puzzle task: (1) odd to even : training with graphs having odd graph sizes and testing on graphs with even numbers of nodes
[Table]
(2) less to more : training on graphs with smaller sizes and testing on graphs with larger sizes
[Table]
and (3) more to less : training on graphs with larger sizes and testing on graphs with smaller.
In the less to more setting
[Table]
we test the model’s ability to use the functions learned from small graphs on more complicated ones
[Table]
whereas the more to less setting evaluates the model’s ability to learn disentangled functions without explicitly seeing them during training.
In our experiments, for the less to more setting, we use sizes less than G/ 2 for training and more than G/ 2 for testing where G is the size of the full graph.
Similarly, for the less to more setting, we use sizes less than G/ 2 for training and more than G/ 2 for testing.
Table 4 reports the NLL for these settings.
The NLL of these models are close to the performance on the models trained on full dataset demonstrating that our model is able to generalize to unseen graph sizes.
[Table]
C ONCLUSION
In this paper, we presented continuous graph ﬂow , a generative model that generalizes the neural message passing in graphs to continuous time.
We formulated the model as an neural ordinary differential equation system with shared and reusable functions that operate over the graph structure.
We conducted evaluation for a diverse set of generation tasks across different domains: graph generation, image puzzle generation, and layout generation for scene graph.
Experimental results showed that continuous graph ﬂow achieves signiﬁcant performance improvement over various of state-ofthe-art baselines.
For future work, we will focus on generation tasks for large-scale graphs which is promising as our model is reversible and memory-efﬁcient.
A A PPENDIX
We provide supplementary materials to support the contents of the main paper.
In this part, we describe implementation details of our model.
We also provide additional qualitative results for the generation tasks: Graph generation, image puzzle generation and layout generation from scene graphs.
A. I MPLEMENTATION D ETAILS
The ODE formulation for continuous graph ﬂow (CGF) model was solved using ODE solver provided by NeuralODE (Chen et al., 2018).
In this section, we provide speciﬁc details of the conﬁguration of our CGF model used in our experiments on two different generation tasks used for evaluation in the paper.
Graph Generation.
For each graph, we ﬁrstly generate its line graph with edges switched to nodes and nodes switched to edges.
Then the graph generation problem is now generating the current nodes values which represents the adjacency matrix in the original graph.
Each node value is binary (0 or 1) and is dequantized to continuous values through variational dequantization, with a global learnable Gaussian distribution as variational distribution.
For our architecture, we use two blocks of continuous graph ﬂow with two fully connected layers in Community-small dataset, and one block of continuous graph ﬂow with one fully connected layer in Citeseer-small dataset.
The hidden dimensions are all 32.
Image puzzle generation.
Each graph for this task comprise nodes corresponding to the puzzle pieces.
The pieces that share an edge in the puzzle grid are considered to be connected and an edge function is deﬁned over those connections.
In our experiments, each node is transformed to an embedding of size 64 using convolutional layer.
The graph message passing is performed over these node embeddings.
The image puzzle generation model is designed using a multi-scale continuous graph ﬂow architecture.
We use two levels of downscaling in our model each of which factors out the channel dimension of the random variable by 2.
We have two blocks of continuous graph ﬂow before each downscaling wth four convolutional message passing blocks in each of them.
Each message passing block has a unary message passing function and binary passing functions based on the edge types – all containing hidden dimensions of 64.
Layout generation for scene graphs.
For scene graph layout generation, a graph comprises node corresponding to object bounding boxes described by { [ x i , y i , h i , w i ] } ni =1 , where x i , y i represents the top-left coordinates, and w i , h i represents the bounding box width and height respectively and edge functions are deﬁned based on the relation types.
In our experiments, the layout generation model uses two blocks of continuous graph ﬂow units, with four linear graph message passing blocks in each of them.
The message passing function uses 64 hidden dimensions, and takes the embedding of node label and edge label in unary message passing function and binary message passing function respectively.
The embedding dimension is also set to 64 dimensions.
For binary message passing function, we pass the messages both through the direction of edge and the reverse direction of edge to increase the model capacity.
A. I MAGE P UZZLE G ENERATION : A DDITIONAL Q UALITATIVE R ESULTS FOR C ELEB A-HQ
Fig.
5 and Fig.
6 presents the image puzzles generated using unconditional generation and conditional generation respectively
. A. L AYOUT G ENERATION FROM S CENE G RAPH : Q UALITATIVE R ESULTS
Fig.
7 and Fig.
8 show qualitative result on unconditional and conditional layout generation from scene graphs for COCO-stuff dataset respectively.
Fig.
9 and Fig.
10 show qualitative result on unconditional and conditional layout generation from scene graphs for Visual Genome dataset respectively.
The generated results have diverse layouts corresponding to a single scene graph.
[Figure]
[Figure] [Figure]
A. G RAPH G ENERATION : A DDITIONAL Q UALITATIVE R ESULTS
Fig.
11 and Fig.
12 present the generated graphs for EGO - SMALL and COMMUNITY - SMALL respectively.
[Figure]
We sample 4 layouts for each scene graph.
The generated results have different layouts except the conditional layout objects in (b), but sharing the same scene graph.
Best viewed in color.
Please zoom in to see the category of each object. A. A NALYSIS : N UMBER OF FUNCTION EVALUATION VS NUMBER OF NODES We analyze the variation in number of function evaluations (NFE) required to solve the ODE as the number of nodes in the graph changes. Refer to Figure 13.
The results interestingly show that the average number of function evaluation does not increase linearly with the increment in number of graph nodes, which would be the case if the variables were independent.
[Figure] [Figure] [Figure]