Invertible Neural Networks for Graph Prediction
Abstract
Graph prediction problems prevail in data analysis and machine learning.
The inverse prediction problem, namely to infer input data from given output labels, is of emerging interest in various applications.
In this work, we develop invertible graph neural network (iGNN), a deep generative model to tackle the inverse prediction problem on graphs by casting it as a conditional generative task.
The proposed model consists of an invertible sub-network that maps one-to-one from data to an intermediate encoded feature, which allows forward prediction by a linear classification sub-network as well as efficient generation from output labels via a parametric mixture model. The invertibility of the encoding sub-network is ensured by a Wasserstein-2 regularization which allows free-form layers in the residual blocks. The model is scalable to large graphs by a factorized parametric mixture model of the encoded feature and is computationally scalable by using
GNN layers.
We study the invertibility of flow mapping based on theories of optimal transport and diffusion process. The proposed iGNN model is experimentally examined on synthetic data, including the example on large graphs, and the empirical advantage is also demonstrated on real-application datasets of solar ramping event data and traffic flow anomaly detection.
Introduction
Graph prediction is an important topic motivated by various applications, e g., protein-protein interaction networks [38], wind power prediction [39] and user behavior modeling in social networks [7].
In the so-called inverse of a graph prediction problem, one would like to infer the input graph nodal features X given an outcome response Y .
Such a problem is of interest in various real-world applications, e.g., molecular design [33] and power outage analysis [41, 1]. The inverse graph prediction problem is the focus of the current work, for which we develop an invertible deep model that can be efficiently applied to graph data.
More formally, The forward prediction problem is to learn the conditional probability of p ( Y | X ) .
This discriminative task can be done by a conventional classification model.
The inverse prediction problem is to learn the conditional probability of p ( X | Y ) and to generate samples X from it. This generative task is challenging when data X is in high dimensional space (e.g., graph nodal features that scales linearly with graph size), where a grid of X can not be efficiently constructed. In addition, in the case of categorical response, the graph label Y assigns one of the K -class labels to each node, which makes the total possible outcomes K N many.
Thus the inverse prediction problem on graph data poses both modeling and computational challenges when scalability to large graphs is needed. In this work, we develop a deep generative model for the conditional generation task of the inverse prediction problem on graphs.
Unlike previous conditional generative models [30, 20, 3, 4, 2], which typically concatenate the prediction label Y with the random code Z as input or rely on curated forms of neural network (NN) layers to ensure invertibility, we propose to encode the input data X one-to-one by an invertible network to an intermediate feature H , from which the label Y
[Figure]
can be predicted using a linear classifier, and, in the other direction, H can also be generated from
Y by a parametric mixture model.
The framework of our model is shown in Figure 1.
Because the general (non-graph) data case can be seen as a special case of graph data where the graph only has one node, we call the proposed model invertible Graph Neural Network (iGNN), as a unified name.
In summary, the contributions of the work are
• We propose a two-step procedure, Y -to- H and H -to- X , to tackle the generative task of inverse prediction problem viewed as a conditional generation problem and develop an invertible flow model consisting of two subnetworks accordingly.
The model is made scalable to graph data by a factorized formulation of the parametric mixture model H | Y and the GNN layers in the invertible flow network between H and X .
• We introduce Wasserstein-2 regularization of the invertible flow network, which is computationally efficient and compatible with free-form layer types including the GNN layers. The effect on preserving invertibility is backed by OT theory and verified in practice.
• We theoretically study the invertibility of flowing mapping based on theories of optimal transport and diffusion process.
• The proposed iGNN model is applied to both simulated and real-data examples, showing improved generative performance over alternative conditional generation models.
Method
Inverse of prediction as conditional generation. The overall framework is to (end-to-end) train a network consisting of two sub-networks: the first sub-network maps invertibly from X to an intermediate representation H , and the second sub-network maps from H to label Y , which is a classifier and loses information.
Specifically, • H - Y classification sub-network.
We model H | Y by a Gaussian mixture model. The parametric form of H | Y contains trainable parameter θ gc , and the generation of H | Y is by sampling the corresponding mixture component accordingly.
The prediction of label Y from H can be conducted by a linear classifier parametrized by θ cc . The trainable parameters in this sub-network are denoted as θ c
= ( θ gc , θ cc ) .
• X - H invertible sub-network.
The invertible mapping from X to H is by a flow ResNet in R d . The sub-network parameters are denoted as θ = { θ l , l = 1 , · · · , L } , where L is the number of residual blocks, and the network mapping is denoted as F θ .
The generation of X | Y is by inversely mapping X
= F − 1 θ ( H ) once H is sampled according to p ( H | Y ) parametrized by θ gc .
The end-to-end training objective of the proposed network can be written as [Equation] [Equation] ) where L g , L c and W are the generative loss, the classification loss and the Wasserstein-2 regular- ization (cf.
Eq.
(7)) , respectively. The scalars µ, γ ≥ 0 are penalty factors. Given n tr many data-label pairs { X i , Y i } , we define L g = 1 n tr (cid:80) n tr i
[Equation]
ℓ g ( X i , Y i ) where
[Equation]
In (2), we parametrize p H | Y by a Gaussian mixture model in R d with a prefixed parameter σ as
[Equation]
In practice, we initialize µ k to be sufficiently separated to ensure non-overlapping supports of Gaussian components, and we can preserve the separation during training via a barrier penalty on the distances ∥ µ k − µ k ′ ∥ .
Figure 2 provides an example of the trained flow in R 2 .
The classification loss is defined as L c = 1 n tr (cid:80) n tr i =1 ℓ c ( F θ ( X i ) , Y i ) , where ℓ c ( H i , Y i ) is the persample K -class classification loss computed by softmax, upon using a linear classifier on F θ ( X i ) . We find the experimental results insensitive to the choice of penalty factor µ and always use µ = 1 .
Scalable conditional generation on graph data. In this subsection, the subscript v indicates graph node v . Suppose the graph ( V, E ) has N nodes in V and the edge set E .
We denote a graph data sample X and a graph label Y as
[Equation] where d ′ is the dimension of node feature. To specify the Gaussian mixture H | Y , we can view X as a vector in R d , d = d ′ N , and label vector Y taking K N many possibilities. Yet, doing so requires specifying a K N -component Gaussian mixture in R d that is not scalable when N is large.
To ensure scalability of iGNN, we introduce a factorized form of H | Y as elaborated below, and propose to use GNN layers in the invertible ResNet (e.g., Chebnet [13] and L3net [11]).
The scalability of our iGNN approach is demonstrated on a larger graph with N = 500 nodes, cf.
Figure 3.
Suppose we have a K -component Gaussian mixture in R d ′ .
We specify the graph H | Y as
[Equation]
that is, the joint distribution of H | Y consists of independent and identical K -component Gaussian mixture distribution of H v | Y v in R d ′ across the graph. As a result, on the graph sample-label pair { X, Y } , the log p ( H | Y ) term in the generative loss (2) can be computed as
[Equation]
Note that the factorized form of H | Y reduces the complexity of modeling H | Y in R d ′ N to that of modeling a K -class mixture model in R d ′ .
Invertible flow network with Wasserstein-2 regularization.
We use an invertible ResNet with L layers to construct the one-to-one mapping between X and H , following the framework of [5]. Let the l -th block residual mapping f ( x, θ l ) be parametrized by θ l .
The ResNet mapping from X to H is H = F θ ( X ) = x L where
[Equation]
The ResNet architecture is illustrated on the right of Figure 1.
The architecture of the residual block is free-form: one can use any layer type inside the residual blocks and even different layer types in different blocks. The Wasserstein-2 ( W 2 ) regularization used in (1) takes the form [Equation] x 0 = X
i [Equation] 7) where x l is defined as in (6). In practice, we numerically verify the invertibility of the trained ResNet in experiments, cf.
Table 1 and Appendix A. We also find a few tens of residual blocks to be enough, including the large graph experiment.
Theory
In this section, we study how the proposed Wassersetein-2 regularization guarantees the invertibility of the flow network.
Theoretically, we consider continuous-time flow induced by a velocity field v ( x, t ) , that is, the continuous-time flow is represented by an initial value problem (IVP) of ODE
[Equation]
The transport in R d is the solution mapping from x (0) =
x to x ( t ) at some time t > 0 .
We refer to our full paper for complete theoretical results and proofs.
3.1 Invertibility of trained flow network
The proposed Wassersetein-2 regularization serves to penalize the transport cost
[Equation]
(9) when the number of residual blocks L is large.
In the special case of K = 1 , we transport from the source density p (the data distribution in R d ) to the target density q , which can be chosen as standard normal N (0 , I d ) .
Our minimizing objective L g + γ W (in population form) is then equivalent to
[Equation]
with some positive scalar ˜ γ .
Compared to the Benamou-Brenier formula [35, 6], the objective (10) relaxes the terminal condition that ρ ( · , 1) =
q to be the KL divergence, which does not change the solution of the optimal v .
The optimal flow induced by the minimizer also gives the Wasserstein geodesic from p to the normal density q , and is smooth when p is a smooth density.
More details of the K = 1 case are given in Section 3.2.
In our problem of conditional generation, when there are K > 1 classes, the limiting continuous time flow differs from the Wasserstein geodesic from ρ ( · , 0) = p ( X | Y ) to ρ ( · , 1) = p ( H | Y ) for fixed Y , but is expected to give a shared flow in R d from X to H , cf.
Figure 2.
Under regularity conditions, one would expect the transport-cost regularized continuous-time flow to also be regular.
As a result, the velocity field v ( x, t ) would have a finite x -Lipschitz constant B on any bounded domain. Divide the time interval [0 , 1] into L time steps, ∆ t = t l +1 − t l = 1 /L .
Then, the transport from x ( t l − 1 ) to x ( t l ) induced by v ( x, t ) on the interval [ t l − 1 , t l ] is invertible when B ∆ t < 1 , which holds when L > B . In this case, also assuming that the solved discrete-time transport map
T l ( x ) : = x
+ f ( x, θ l ) is close to that induced by v ( x, t ) , one can expect the invertibility of T l in each residual block, and then the composed transport F θ over L blocks is also invertible.
3.2 The special case of K =
For the continuous-time flow, the transport mapping by the IVP (8) is invertible as long as (8) is well-posed.
The normalizing flow from p to normal q is typically not unique.
We introduce two constructions here that are related to the proposed Wasserstein-2 regularization.
(i) Flow by Benamou-Brenier formula.
Consider (8) on t ∈ [0 , 1] , the flow F maps from x = x (0) to x (1) .
Let ρ ( · , t ) be the density of x ( t ) , ρ ( · , 0)
=
p and ρ ( · , 1) = F # p , the push-forward final density by F .
For the flow induced by the optimal v ( x, t ) in (10) which is equivalent to the Benamou-Brenier formula, the regularity of v and ρ follows from classical OT theory:
Proposition 3.1 ([9, 36]) .
Suppose p is smooth on R d with finite moments, then the optimal velocity field v ( x, t ) that minimizes (10) is smooth, the induced IVP (8) is well-posed on R d × [0 , 1] and the flow mapping is smooth and invertible. (ii) Flow by Fokker-Planck equation. We use the Fokker-Planck equation of stochastic process to provide another theoretical construction of the invertible flow. Because q is standard normal, the stochastic process is an Ornstein-Uhlenbeck (OU) process in R d , for which the Fokker-Planck equation can be written as
[Equation]
where ρ ( x, t ) represents the probability density of the OU process at time t .
The Liouville equation of (8) is ∂ t ρ = −∇ · ( ρv ) , where ρ ( x, t ) represents the density of x ( t ) .
Comparing to (11), we see that the density evolution can be made the same if the velocity field v ( x, t ) is set to satisfy
[Equation]
The smoothness of v follows from the explicit expression of ρ ( x, t ) as the solution of (11). Proposition 3.2. Let ρ ( x, t ) be the solution to (11) from ρ ( x, 0) = p , then the IVP (8) induced by velocity field v ( x, t ) as in (12) is well-posed on R d × (0 , T ) for any T > 0 and the flow mapping is smooth and invertible.
While theoretically the density ρ ( x, t ) in (11) converges to the normal equilibrium q in infinite time, the convergence is exponentially fast [8].
Thus the transported density F # p = ρ ( · , T ) for a finite T ∼ log(1 /ϵ ) can be ϵ -close to q .
Experiment
We experimentally examine the proposed iGNN model on simulated data and real graph data (solar ramping event data and traffic flow anomaly detection). We also compare spectral and spatial GNN layers on a simulated example.
The experimental setup is provided in Appendix A.1, and further details are in Appendix A.2.
Code is available at https://github.com/hamrel-cxu/
Invertible-Graph-Neural-Network-iGNN .
We consider three simulated examples in this section. The first considers non-graph Euclidean vector data, and the second and the third consider graph data on small and large graphs.
Non-graph data in R 2 .
The dataset contains a Gaussian mixture of eight components with four classes, where each X | Y is further divided into two disjoint Gaussian distributions in R 2 . Figure A.2 compares our generative results with cINN-MMD, where both methods can generate data that are reasonably close to X | Y at each Y .
Data on a small graph.
At each node v , Y v ∈ { 0 , 1 } and features X v ∈ R 2 , thus the graph node label vector Y ∈ { 0 , 1 } 3 .
Detailed data-generating procedures can be found in Appendix A.1.
In this example, iGNN yields comparable generative performance with cINN-MMD, see Figure A. and more results in Appendix A.2.
quality and data details are described in Section 4.
[Table] [Figure]
Data on a large graph.
We consider a 503-node chordal cycle graph [27], which is an expander graph. We design binary node labels and let node features X | Y ∼ N ( µ Y , Σ Y ) , where X v ∈ Rand the mean µ Y and covariance matrix Σ Y contain graph information. Detailed data-generating procedures can be found in Appendix A.1. Because enumerating all values of Y is infeasible, we randomly choose 50 values of outcome Y , each of which has 50% randomly selected node labels to be 1.
To visualize the generative performance of iGNN, we compare the covariance of true and generated data restricted to subgraphs. Specifically, we plot the covariance matrix of modelgenerated data ˆ X | Y and true data X | Y on sub-graphs produced by 1 or 2-hop neighborhoods of a graph node. Figure 3 shows the resemblance between learned and true covariance matrices on different neighborhoods on the graph. [Figure]
We apply the iGNN model to two graph prediction data in real applications. The inverse prediction problem is formulated as a conditional generation task.
Solar ramping events data.
Consider the anomaly detection task on California solar data inand 2018, which were collected in ten downtown locations representing nodes.
Each node records non-negative bi-hourly radiation recordings measured in Global Horizontal Irradiance (GHI).
After pre-processing, features X t ∈ R 10 × 2 denote the average of raw radiation recordings every 12 hours in the past 24 hours, and response vectors Y t ∈ { 0 , 1 } 10 contain the anomaly statusy.
Figure 4 shows that the learned conditional distribution ˆ X | Y by iGNN model closely resembles that of the true data X | Y , and outperforms the generation of cINN-MMD.
The quantitative evaluation is given in Table 2, which shows that iGNN has comparable or better performance than the alternative approaches (smaller test statistics indicate better generation). The table also shows that cINN-Flow performs significantly worse than cINN-MMD and iGNN on this example, which is consistent with the visual comparison of ˆ X | Y (not shown).
Lastly, Figure 5 shows the predictive capability of iGNN: given a test node feature matrix X , we can compute P ( Y i = 1 | X ) for node i using the
[Figure] [Table]
trained linear classifier on F Θ ( X ) .
The predicted probabilities learned by the model are consistent with the true nodal labels, and provide more information than the binary prediction output.
Traffic flow anomalies. We study the anomaly detection task on Los Angeles traffic flow data from April to September 2019.
The whole network has 15 sensors with hourly recordings. Features X t ∈ R 15 × 2 denote the raw hourly recording in the past two hours, and response vectors Y t ∈ { 0 , 1 } 15 contain the anomaly status of each traffic sensor. The graph topology is shown in Figure 6(a), along with the raw input features in R 2 (over all graph nodes). The generative performance by iGNN resembles the ground truth, as shown in Figure 6(b), and is better than the generative performance by cINN-MMD in (c).
The quantitative evaluation metrics also reveal the better performance of iGNN over alternative baselines, cf.
Table 2.
4.3 Comparison of GNN layers
We examine the empirical performance of different GNN layers in learning the flow of graph data.
Here we provide an example on a three-node graph where spectral graph filters lack expressiveness due to constructed symmetry.
Example 1 (Insufficient expressiveness of spectral graph filters) .
Consider a graph with three nodes { 1 , 2 , 3 } and two edges { (1 , 2) , (2 , 3) } between nodes.
Self-loops at each node are also inserted.
For nodal features X ∈ R 3 , assume X ∼ N (0 , Σ) . Let the covariance matrix Σ and permutation matrix π take the form Σ := (cid:34) [Figure] [Figure]
[Figure]
ρ
[Figure]
(cid:35) ,
[Figure]
(cid:34)
[Figure]
[Figure]
[Figure] [Figure] One can see that π Σ π T ̸ = Σ if ρ ̸ = ρ 1 . On the other hand, πAπ T =
A at this chosen permutation π , so that any spectral graph filter f ( A ) as a matrix function of A satisfies that πf ( A ) π T = f ( A ) . As a result, f ( A ) for any f will make an O (1) error in approximating Σ and thus fail to generate correct samples. Because (possibly normalized) graph Laplacian is either a polynomial of A or preserves the same symmetry pattern as A , the issue happens with any spectral convolutional filter.
We simulate data based on the three-node graph in Example 1. Figure 7 compares the generative performance of iGNN by using ChebNet and L3Net layers.
In this example, iGNN with ChebNet layers fails to learn the conditional distribution X | Y . The plot (b) shows that the learned ˆ X | Y has a symmetry of nodes 1 and 3, which explains the cause of insufficiency. Meanwhile, iGNN with L3Net layers yields satisfactory performance by having sufficient model expressiveness.
Related works
Generative deep models and normalizing flow.
At present, generative adversarial networks (GAN) [15, 19] and variational auto-encoders (VAE) [23, 24] are two of the most popular frameworks that have achieved various successes [29, 40, 26]. However, they also suffer from clear limitations such as notable difficulties in training, such as mode collapse [32] and posterior collapse [28]. On the other hand, normalizing flows (see [25] for a comprehensive review) estimate arbitrarily complex densities via the maximum likelihood estimation (MLE), and they transport original random features X into distribution that are easier to sample from (e.g., standard multivariate Gaussian) through invertible neural networks.
Flow-based models can be classified into two broad classes: the discretetime models (some of which include coupling layers [14], autoregressive layers [37] and residual networks [5, 10]), and the continuous-time models as exemplified by neural ODE [17, 31].
Most normalizing flow methods focus on unconditional generation with little development in a conditional generation.
In addition, to achieve numerically reliable training, regularization of the density transport trajectories in flow networks are necessary but remain a challenge.
Conditional generation networks.
The conditional generation versions of GAN (cGAN) have been studied in several places [30, 20], where the prediction outcome Y (one-hot encoded as e Y ) are typically concatenated with random noise Z and taken as input to the generator network, as illustrated in the left of Figure 1.
The one-hot encoding of categorical Y concatenated with Gaussian Z poses challenges in training cGAN models, in addition to known issues of their unconditional counterparts, such as mode collapse, posterior collapse, and failures to provide exact data likelihood.
When the prediction label Y is lying on a graph having N nodes, the one-hot coding will increase up to
O ( N ) more coordinates to the input ( Z, e Y ) , which significantly increases the model complexity and computational load with large graphs.
Conditional invertible neural network (cINN) model was developed in [3] for analyzing inverse problems.
The model inherits the approach of cGAN models to concatenate one-hot encoded prediction label e Y with normal code Z while using Real-NVP layers [14] to ensure neural network invertibility.
In terms of training objective, [3] proposed to use maximum mean discrepancy (MMD) losses to encourage both the matching of the input data distribution and the independence between label Y and normal code Z .
We call the method in [3] cINN-MMD.
Replacing the MMD losses with a flow-based objective, [4, 2] extended the invertible network approach in [3] and applied to image generation problems.
In the models in [4] and [2], which we call cINN-Flow and cINN-Flow+ respectively, the inputs to the Real-NVP layers contain encoded information of the prediction label Y so as to learn label-conditioned generation.
Comparison to spectral normalization.
iResNet [5] proposed spectral normalization to ensure the invertibility of each residual block.
Given a weight matrix W ∈ R C × C ′ in a fully-connected layer, the method first computes an estimator ˜ σ of the spectral norm ∥ W ∥ 2 by power iteration [16], and then modify the weight matrix W to be cW/ ˜ σ if c/ ˜ σ < 1 , where c < 1 is a pre-set scaling parameter.
It was proposed to apply the procedure to all weight matrices in all ResNet blocks in every stochastic gradient descent (SGD) step with mini-batches.
When the number of blocks L is large, this involves expensive computation, especially if the hidden layers are wide, i.e., C and C ′ being large.
In addition, while spectral normalization of fully-connected layers together with contractive nonlinearities (e.g., ReLU, ELU, Tanh) ensures invertibility, it may not be directly applicable to other layer types, e.g., GNN layers.
In contrast, the proposed Wasserstein-2 regularization are obtained from the forward passes of the residual blocks on training samples without additional computation.
It is also generally compatible with free-form neural network layer types. Acknowledgement The work is supported by NSF DMS-2134037. C.X. and Y.X. are supported by an NSF CAREER
Award CCF-1650913, NSF DMS-2134037, CMMI-2015787, DMS-1938106, and DMS-1830210.
X.C. is partially supported by NSF, NIH and the Alfred P. Sloan Foundation.
A Additional experimental details A.1 Experimental set-up A.1.1 Computation of log det
To compute the log determinant in (2), we adopt the following unbiased log determinant approximation technique as proposed in [10].
Let F θ ( x ) = x
+ f θ ( x ) denote the output from a generic ResNet block with parameter θ .
First, observe that for any input x , log | det J F θ ( x ) | = tr (log J F θ ( x )) because the matrix J F θ ( x ) is non-singular.
We thus have tr (log J F θ ( x )) =
tr (log( I + J f θ ( x ))) .
As a result, the trace of the matrix logarithm can be expressed as
[Equation]
Based on (13), which takes infinite time to compute, we can obtain an unbiased estimator in finite time based on the “Russian roulette“ estimator approach [21, 10].
For a ResNet as a concatenation of L ResNet blocks, the approximation is applied to each block and summed over all blocks.
Lastly, to speed up gradient computation of the approximation, we further adopt memory efficient backpropagation through the early computation of gradients [10].
A.1.2 Baselines and evaluation metric.
We consider three competing conditional generative models:
• Conditional generative adversarial network (cGAN) [20].
• Conditional invertible neural network with maximum mean discrepancy (cINN-MMD)
• Conditional invertible neural network using normalizing flow (cINN-Flow) [4].
To quantify performance, we measure the difference between two distributions ( X | Y versus ˆ X | Y at different Y ) by kernel maximum mean discrepancy (MMD) [18] and energy statistics [34].
Details of the MMD and energy statistics metrics are contained in Appendix A.1.4.
We also provide qualitative comparison by visualizing the generated data samples.
A.1.3 Data and ResNet architecture.
In the examples of graph data, the number of graph nodes ranges from 3 to 500.
All graphs are undirected and unweighted, with inserted self-loops.
Regarding the ResNet block layer type: for non-graph data, we use 2 fully-connected hidden layers of 64 neurons in all ResNet blocks.
For graph data, in each residual block, we replace the first hidden layer to be a GNN layer (either ChebNet or L3Net layer); the second layer is a shared fully-connected layer that applies channel mixing across all the graph nodes (which can be viewed as a GNN layer with identity spatial convolution).
The activation function is chosen as ELU [12] or LipSwish [10], which have continuous derivatives.
The network is trained end-to-end with the Adam optimizer [22].
A.1.4 Model evaluation metrics
MMD statistics metric.
Given two sets of samples X = { x 1 , . . .
, x n } , X ′ = { x ′ 1 , . . .
, x ′ n } of same sample size n , the MMD two-sample statistic between X and X ′ is defined as
[Equation]
[Equation]
where we use the radial basis kernel k ( x, x ′ )
=
exp( − α ∥ x − x ′ ∥ 2 ) with α = 0 . 1 . For the K -class conditional distribution X | Y , where we denote by { X | Y = k } the set of samples { x i : y i = k } ni =1 , the overall MMD statistic is defined using (14) as MMD =
[Equation]
n
[Equation]
− 1 −
X F − 1Θ ( F Θ ( X )) − 1 −
X F Θ ( F − 1Θ ( X ))
{+(a) In-distribution sample X 0 .0 .
+}
{+X F − 1Θ ( F Θ ( X )) 0 .0 .0 .0 .1 . 
+}
X F Θ ( F − 1Θ ( X )) (
[Figure]
of-distribution sample X
Note that on graph data where Y concatenates all nodal labels, the summation is over all types of Y (up to K N many).
Energy statistic metric.
Given two sets of samples X = { x 1 , . . .
, x n } , X ′ = { x ′ 1 , . . . , x ′ n } , The energy statistic under ℓ 2 norm is defined as [Equation] For K -class conditional distribution X | Y , the weighted energy statistics is defined using (16) as
Energy = [Equation] Computation of the weighted statistics on graph data is identical to that of the weighted MMD statistics on graph. Model invertibility error.
[-= (cid:80) 2 k =
-]We see from Figure A.1 that iGNN under the Wasserstein-2 regularization ensures model invertibility up to very high accuracy. A.1.5 Construction of simulated graph data We describe the construction of simulated graph data on small graphs (corresponding to Figuresand A.4). Each node has a binary label so that Y ∈ { 0 , 1 } 3 is a binary vector.
Conditioning on a specific binary vector Y out of the eight choices, the distribution of X | Y is defined as [Equation] where P A : =
D − 1 A A is the graph averaging matrix. The distribution of Z | Y is independent but non-identically distributed over nodes. On node 0 and 2, we let it be rotated and shifted noisy two moons, whereas on node 1, we let it be a Gaussian mixture in R 2 . On large graphs (corresponding to Figure 3), each node has a binary label so that Y ∈ { 0 , 1 } 503 is a binary vector.
Conditioning on a specific binary vector Y out of the eight choices, the distribution
[Figure] [Figure] [Figure] of X
| Y is defined as
[Equation]
where R denotes a counter-clockwise rotation matrix for 90 degrees and P A : = D − 1 A A is the graph averaging matrix. We choose δ = 0 .
2 so that a soft graph averaging is applied to the hidden variables
Z | Y .
A.2 Additional experimental results
This subsection contains experimental results to augment those in the main text.
In particular, • We show conditional generation performance by iGNN and cINN-MMD on non-graph data in Figure A.2. • We show the confidence region on the three-moon non-graph conditional generation data in Figure A.3.
The generative quality is presented in Figure 2 in the main text.
• We compare the generative quality of iGNN and cINN-MMD on simulated three-node graph conditional generation data in Figure A.4. [Figure]