Paper:
Constrained Generation of Semantically Valid Graphs via Regularizing Variational Autoencoders
Deep generative models have achieved remarkable success in various data domains, including images, time series, and natural languages. There remain, however, substantial challenges for combinatorial structures, including graphs. One of the key challenges lies in the difficulty of ensuring semantic validity in context. For example, in molecular graphs, the number of bonding-electron pairs must not exceed the valence of an atom; whereas in protein interaction networks, two proteins may be connected only when they belong to the same or correlated gene ontology terms. These constraints are not easy to be incorporated into a generative model. In this work, we propose a regularization framework for variational autoencoders as a step toward semantic validity. We focus on the matrix representation of graphs and formulate penalty terms that regularize the output distribution of the decoder to encourage the satisfaction of validity constraints. Experimental results confirm a much higher likelihood of sampling valid graphs in our approach, compared with others reported in the literature.

Review:
This paper investigates a method for learning deep generative models of graphs, subject to constraints of the types that arise in many practical situations of interest. Rather than specify the model in such a way that it explicitly encodes valid graphs, instead a Lagrangian is formulated which penalizes generation of graphs which violate the constraints.   In a VAE setting, the constraints themselves only apply to the generated graphs (x) not the latent variables (z). To ensure the constraints hold over the entire domain, the constraints are evaluated via Monte Carlo integration with values separately simulated from the prior p(z), rather than the values drawn from the approximating posterior q(z|x).  The overall idea is fairly straightforward — Lagrange multipliers are the standard approach for converting constrained to unconstrained optimization problems. The most interesting aspect of this paper is in Theorem 1, which makes explicit that regularization of the VAE objective in this manner is appropriate. However, the exact form used is not justified particularly well in the main text. ** In particular: why in eq (6) would one consider the marginalization over the squared constraint, rather than the marginalization itself? Is this necessary for Theorem 1 to hold? The use of a ramp function in equation (7) seems a bit of a hack. ** If this were addressed more clearly, or better motivated, I would adjust my score upwards.  There is also an overloading of notation which becomes a bit confusing, for the functions g_i(…, …). In equations (4) and (6), these are functions of x and z, while in e.g. equation (7) g_i is a function of theta and z. It would be good to separate the process which transforms a sampled z into a graph x (a process parameterized by theta) from the definitions of the constraints themselves, which typically seem to be only a function of the simulated x, not directly of z (the constraints are on graph properties, not on z).  The experimental validation is fairly minimal, and does not compare against other graph generation methods, but rather only against VAEs which produce SMILES strings. I would think there should be some baselines for graph-based methods available for some of the proposed tasks. (For example, would it be possible to define exponential random graph models (ERGMs) which would include the constraints explicitly as likelihood terms, at the expense of easily simulating graphs?) As an example, the arXiv paper https://arxiv.org/abs/1805.09076 considers similar molecule generation tasks, but includes a couple explicitly graph-based baselines.