T OP -N: E QUIVARIANT SET AND GRAPH GENERATION WITHOUT EXCHANGEABILITY
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider one-shot probabilistic decoders that map a vector-shaped prior to a distribution over sets or graphs.
These functions can be integrated into variational autoencoders (VAE), generative adversarial networks (GAN) or normalizing ﬂows, and have important applications in drug discovery.
Set and graph generation is most commonly performed by generating points (and sometimes edge weights) i.i.d.
from a normal distribution, and processing them along with the prior vector using Transformer layers or graph neural networks.
This architecture is designed to generate exchangeable distributions (all permutations of an output are equally likely) but it is hard to train due to the stochasticity of i.i.d.
generation.
In this work, we propose a new deﬁnition of equivariance and show that exchangeability is in fact unnecessary in VAEs and GANs.
We then introduce
Top-n , a deterministic, non-exchangeable set creation mechanism which learns to select the most relevant points from a trainable reference set.
Top-n can replace i.i.d.
generation in any VAE or GAN – it is easier to train and better captures complex dependencies in the data.
Top-n outperforms i.i.d.
generation by 15% at SetMNIST reconstruction, generates sets that are 64% closer to the true distribution on a synthetic molecule-like dataset, and is able to generate more realistic molecules when trained on the classical QM9 dataset.
With improved foundations in one-shot generation, our algorithm contributes to the design of more effective molecule generation methods.
I NTRODUCTION
In recent years, many architectures have been proposed for learning vector representations of sets and graphs.
Sets and graphs are unordered and therefore have the same symmetry group, which offers a fruitful theoretical framework for the development of new models (Bronstein et al., 2021).
The opposite problem, i.e., learning to generate sets and graphs from a low-dimensional vector prior, has been less explored.
Yet, it has important applications in drug discovery, for example, where neural networks can be used to sample from the space of stable drug-like molecules.
As set and graph generation are very similar problems, we generally treat them together in this paper 1 .
In particular, one can navigate between set and graph representations using Set2Graph functions (Serviansky et al., 2020) or graph neural networks.
There are two main classes of probabilistic decoders for sets: recursive and one-shot.
Recursive generators are conceptually simpler, since they add nodes, edges or substructures one by one (Liu et al., 2018;
Liao et al., 2019; Sun et al., 2020).
They are however slow and introduce an order in the points that does not exist in the data.
In this paper, we instead focus on one-shot generation, which allows for more principled designs.
These probabilistic decoders can be incorporated into any architecture (generative adversarial networks, variational autoencoders, normalizing ﬂows or diffusion models) and trained to generate sets from a latent vector.
By deﬁnition, one-shot generative models for sets feature a layer that maps a vector to an initial set.
We group all layers until this one into a creation module , and the subsequent layers into an update module .
The design of the update is well understood: as it maps a set to another set, any permutation equivariant neural network suits this task.
In contrast, the creation step poses two major challenges, which are i) the need to generate sets of various sizes, and ii) the generation of different points from a single prior vector, which cannot be done with a permutation equivariant function.
Set creation is typically performed in two steps: ﬁrst, points are drawn i.i.d.
from a low-dimensional normal distribution, and then the latent vector is appended to each point.
Architectures that use this design can generate any number of points, and decouple the number of generated points from the number of trainable parameters in the model.
Furthermore, when combined with a permutation equivariant update module, they produce exchangeable distributions, a property which is commonly held as the equivalent of permutation equivariance for generative models.
However, the stochasticity of i.i.d.
creation makes learning difﬁcult, as the network has to learn to reconstruct a set for any combination of values drawn from the base distribution (Krawczuk et al., 2021).
In this work, we ﬁrst extend the usual deﬁnition of equivariance.
Given a neural network architecture F ⇥ parametrized by ⇥ and a loss function l , we say that the pair ( F, l ) is equivariant to the action of a group if the dynamics of ⇥ trained on gradient descent on l are independent of the group elements used to represent the data.
Using this deﬁnition, we provide sufﬁcient conditions for equivariance and show that exchangeability is actually not needed for equivariance in GANs and VAEs, thus explaining why i.i.d.
generation is empirically not superior to other approaches.
We then propose a new set creation method, called Top-n creation , which relies on a trainable reference set where each of the point has a representation r i 2 R c and an angle  i 2 R a .
To generate a set with n elements, we select the n points whose angles have the largest cosine with the latent vector.
In order to make this process differentiable, we build upon the Top- K pooling mechanism (Gao & Ji, 2019) initially proposed for graph coarsening.
Top-n can eventually be integrated in any
VAE or GAN to form a complete generative model for sets or graphs.
As it is deterministic, it is easier to train than i.i.d.
generators; in contrary to MLPs, it can extrapolate to larger sets than those seen during training.
We benchmark our creation method on both set and graph generation tasks: it reveals that Top-n is able to reconstruct the data more accurately on a set version of MNIST, to extrapolate to larger sets and ﬁt more closely the true distribution (on a dataset of synthetic molecule-like structures in 3D), and to generate realistic graphs (on the QM9 chemical dataset).
Overall, our work provides new theoretical foundations for one-shot set and graph generation and a more efﬁcient method that contributes to the development of effective learning-based methods for drug discovery.
T HE ONE - SHOT SET GENERATION PROBLEM
We consider the problem of learning a probabilistic decoder f that maps latent vectors z 2 R l to multi-sets 2 X
= { x 1 , ..., x n } where each x i 2 R d for some varying number of points n .
Given sample sets from an unknown distribution D , f should be such that, if z is drawn from a prior distribution p Z ( z ) , then the push-forward measure f # ( p Z ) (i.e., the law of f ( z ) ) is close to D .
In practice, representing sets is not convenient on standard hardware, and sets are internally represented by matrices X 2 X =
S n 2 N R n ⇥ d where each row represents a point x i 2 R d .
Algorithms that return a set implicitly assume the use of a function mat-to-set that maps a matrix X to the corresponding set X .
Existing architectures for one-shot set or graph generation use one of the two generation mechanisms described in Figure 1.
First of all, a set cardinality n has to be sampled.
Two strategies have been proposed:
• Most of the literature assumes that n is known during training.
The distribution of the number of points in the training data is computed, and at generation time, a variable is sampled from this multinomial distribution.
This method assumes that the latent vector z is independent of n , so that the generative mechanism writes p ( X| n, z ) p ( n ) p ( z ) .
• Kosiorek et al.
(2020) instead proposed to learn the value of n from the latent vector using a MLP.
This layer is trained via an auxiliary loss, but the predicted value is used only at generation time (the ground truth cardinality is used during training).
The generative model is in this case p ( X| z , n ) p ( n | z ) p ( z ) , i.e., z and n are not independent anymore.
Once n is sampled, one-shot generation models can formally be decomposed into several components: a ﬁrst function g (that we call creation function ) maps the latent vector to an initial set
X 0 2 R n ⇥ c .
This function is usually simple, and is therefore not able to model complex dependencies within each set.
For this reason, X 0 may then be reﬁned by a second function h that we call update , so that the whole model can be written f = mat-to-set  h  g .
We now review the parametrizations that have been proposed for the creation and update modules.
2. M ETHODS FOR SET CREATION
MLP based generation Early works (Achlioptas et al.
; Zhang et al.
(2019b) for sets, Guarino et al.
(2017); De Cao & Kipf (2018); Simonovsky & Komodakis (2018) for graphs) proposed to learn a MLP from R l to R n max c , where n max is the largest set size in the training data.
The output vector is then reshaped as a n max ⇥ c matrix, and masked to keep only the ﬁrst n rows.
MLPs ignore the symmetries of the problem and are only trained to generate up to n max points, with no ability to extrapolate to larger sets.
Despite these limitations, they often perform on par with more complex methods, and still constitute the standard way to generate small graphs (Madhawa et al., 2020;
Mitton et al., 2021).
We explain in Section 3 this surprising phenomenon.
Random i.i.d. generation Along with MLP based generation, the most popular method for set and graph creation is to draw n points i.i.d.
from a low dimensional normal distribution, and to concatenate the latent vector to each sample (K¨ohler et al., 2020;
Yang et al., 2019b; Kosiorek et al., 2020; Stelzner et al., 2020; Satorras et al., 2021; Zhang et al., 2021; Liu et al., 2021).
The main advantage of random i.i.d.
creation is that it does not constrain the number of points that can be generated.
Furthermore, it is exchangeable, i.e., all permutations of the rows of X 0 are equally likely.
This property is widely considered as the equivalent of equivariance for generative models (Yang et al., 2019a; Biloˇs & G¨unnemann, 2021;
Kim et al., 2021; K¨ohler et al., 2020; Li et al., 2020).
Unfortunately, i.i.d. generation makes training very difﬁcult.
Generative models based on i.i.d.
creation indeed have two stochastic components: the prior vector that is sampled through the reparametrization trick (Kingma & Welling, 2013) and the values for the initial set drawn from the normal distribution – both components make it harder to generate realistic sets or reconstruct the training data (Krawczuk et al., 2021), which we also observe experimentally (Appendix D).
First-n creation While MLP and random generation are the most common methods for set creation, Zhang et al. (2019a) and Krawczuk et al.
(2021) use another design: they always start from the same learnable set X ref 2 R n max ⇥ c , and concatenate the latent vector to each point of this set.
Similarly to MLP based creation, they then mask this matrix to keep only the ﬁrst n rows: we therefore call this method First-n creation.
First-n converges much faster than with i.i.d.
generation (Krawczuk et al., 2021), but the network is only trained to generate up to n max points and has no ability to extrapolate to larger sets.
Furthermore, selecting the ﬁrst n rows of the reference set X ref introduces a bias because the ﬁrst rows are selected more often than the last ones.
Creation methods for graph generation For graph generation, edge weights (or edge features) also need to be learned.
To the best of our knowledge, First-n creation has not been used for this purpose yet, and the weights are generated either by a MLP or by sampling i.i.d.
normal entries.
Alternatively, Bresson & Laurent (2019) and Krawczuk et al.
(2021) proposed to generate a set at creation time, and use a Set2Graph update function in order to learn the graph adjacency matrix.
2. M ETHODS FOR SET UPDATE
Since all creation methods except MLPs can only generate very simple sets and adjacency matrices, additional layers are usually used to reﬁne these objects – we gather these layers into the update module.
Because these layers map a set or a graph to another set/graph, the update module falls into the standard framework of permutation equivariant representation learning and all existing equivariant layers can be used: Deep sets (Zaheer et al., 2017), self-attention (Vaswani et al., 2017; Lee et al., 2019), Set2Graph (Serviansky et al., 2020), graph neural networks (Battaglia et al., 2018) or higher-order neural networks (Maron et al., 2018).
Recently, Transformer layers have constituted the most popular method for set and graph update (Bresson & Laurent, 2019; Kosiorek et al., 2020;
Stelzner et al., 2020; Krawczuk et al., 2021) – we also use such layers in our experiments.
A N EQUIVARIANCE PERSPECTIVE ON SET AND GRAPH GENERATION
In this section, we analyse one-shot set and graph generation from the perspective of permutation equivariance.
As set and graphs are unordered, the symmetric group S =
S n 2 N ⇤ S n containing all permutations is a symmetry of these tasks.
A permutation ⇡ 2 S n acts on a n ⇥ n matrix A by permuting its rows and columns (which we write ⇡ .
A = ⇡ A ⇡ T ), on a n ⇥ c matrix by permuting its rows ( ⇡ .
X = ⇡ X ), and leaves a vector z 2 R h unchanged ( ⇡ .
z = z ).
This symmetry constitutes a useless factor of variation in the data that should be factored out in the latent space.
It is achieved when the latent representation z is invariant to the action of S ( ⇡ .
z = z ).
In discriminative models, symmetries are accounted for when a neural network f is equivariant to the action of the permutation group (Kondor, 2008), which writes ⇡ .f
( X ) =
f
( ⇡ . X ) .
This deﬁnition can unfortunately not be used for set and graph generation from a latent vector.
Indeed, imposing ⇡ .f
( z ) =
f
( ⇡ . z ) =
f ( z ) would only allow for trivial solutions where all rows are equal, which is too restrictive.
Another deﬁnition, that provides more relaxed conditions, is therefore needed for taking symmetries into account.
For this purpose, we propose a deﬁnition called ( F, l ) -equivariance.
Our proposition is based on the observation that equivariant models for discriminative tasks are almost always used along with invariant loss functions.
For example, the l 2 loss
||
Y
ˆ Y
|| 2 is commonly used to learn the future state of a n -body system, but not the l 1 loss ||
Y  ˆ Y || 1 , as it is not rotation invariant.
Formally, if F ⇥ = { f ✓ : X !
Y ; ✓ 2 ⇥ } is an hypothesis class of G - equivariant functions from X to Y (for example a neural architecture parametrized by ✓ ), then the loss functions l should satisfy
[Equation]
When l is chosen accordingly, the gradients with respect to the parameters satisfy r ✓ l ( f ( g. X ) , g. Y ) =
r ✓ l ( f ( X ) , Y ) , i.e., each parameter update is independent of the group elements that are used to represent X and Y .
It follows that the training dynamics as a whole become independent of the group elements used to represent the data, which makes data augmentation unnecessary.
We therefore propose the following deﬁnition:
Deﬁnition 1 ( ( F, l ) -equivariance) .
Consider an hypothesis class F ⇥ ⇢ Y X , a group G that acts on X and Y and a loss function l deﬁned on Y .
We say that the pair ( F ⇥ , l ) is equivariant to the action of
G if the dynamics of ✓ 2 ⇥ trained with gradient descent on l do not depend on the group elements that are used to represent the training data.
With our deﬁnition, the use of an equivariant architecture and an invariant loss is a sufﬁcient condition for ( F, l ) -equivariance in discriminative settings.
As for standard generative architectures for sets and graphs, we can derive the following sufﬁcient conditions (proofs are given in Appendix A):
Lemma 1.
Sufﬁcient conditions for ( F, l ) -equivariance:
GANs: if F is a GAN architecture with a permutation invariant discriminator, and l the standard GAN loss, then ( F, l ) is permutation equivariant.
No constraint is imposed on the generator.
Autoencoders, VAEs, DSPN: if F is an encoder-decoder architecture with a permutation invariant encoder, and the reconstruction loss l satisﬁes 8 ⇡ 2 S , l ( ⇡ .
X , ˆ X ) =
l ( X , ˆ X ) , then ( F, l ) is permutation equivariant.
No constraint is imposed on the decoder function.
Normalizing ﬂows: if F is an architecture such that the set creation yields an exchangeable distribution, the update is permutation equivariant and invertible, and p ✓ denotes the model likelihood, then ( F,  log p ✓ ) is permutation equivariant (proved in K¨ohler et al.
As desired, ( F, l ) -equivariance is a notion of equivariance that does not impose the equivariance of the probabilistic decoder in generative architectures.
Furthermore, for GANs and VAEs, we observe that the set generator does not need to yield exchangeable distributions either.
To understand why exchangeability is not needed, recall that in GANs and VAEs a mat-to-set function is implicitly applied to the model output: a method that generates matrices that are always permuted in the same way is therefore equivalent to one that generates exchangeable matrices (as the corresponding sets are the same).
In other words, the fact that the output of the model is not a matrix but a set is an assumption, not something that needs to be proved 3 .
This observation explains why random i.i.d.
generation empirically does not outperform non-exchangeable set creation methods such as MLPs and First-n (Krawczuk et al., 2021).
We also note that the constraints of Lemma 1 are satisﬁed by most existing architectures, including early ones (Simonovsky & Komodakis, 2018; De Cao & Kipf, 2018; K¨ohler et al., 2020).
For example, the constraint on the loss for VAEs ( 8 ⇡ 2 S , l ( ⇡ .
X , ˆ X ) =
l ( X , ˆ X ) ) is satisﬁed by the two loss functions commonly used for sets, namely Chamfer loss and the Wasserstein-2 distance, deﬁned as
[Equation]
where  is the set of couplings (i.e., bistochastic matrices) between X and X 0 .
While efﬁcient algorithms exist for computing the Wasserstein distance (Cuturi, 2013; Feydy et al., 2019), the equation l ( ⇡ .
X , ˆ X ) =
l ( X , ˆ X ) may be difﬁcult to satisfy in other settings: matching graphs up to permutations, or sets up to the SE (3) group, leads to difﬁcult problems for which no polynomial time algorithm is known (M´emoli, 2007).
In these settings, the design of VAE is harder and other architectures may be better suited.
Overall, the notion of ( F, l ) -equivariance provides justiﬁcations to the design of standard architectures for both generative and discriminative tasks.
Contrary to common belief, it also appears that exchangeability not needed to build equivariant GANs and VAEs for sets and graphs.
In the following section, we therefore design a new creation mechanism without worrying about the model exchangeability.
T HE T OP - N CREATION MECHANISM
We have seen in Section 2 that existing set creations methods suffer from important limitations:
i.i.d. creation is slow to train due to the random initial points, while MLPs and First-n use a ﬁxed mask to select the correct number of points and cannot extrapolate to larger sets.
In order to solve these limitations, we propose a new method called Top-n creation, which is summarized in Figure 2.
Similarly to First-n, Top-n also uses a reference set, but in Top-n this set can have an arbitrary size n 0 .
Each point in this set is a pair (  , r ) : the angle  2 R a is used to decide when to select the point, and r 2 R c contains the representation of the point.
Given a latent vector z 2 R l ⇥ 1 , a reference set made of angles  2 R n a and representations R 2 R n 0 ⇥ c and matrices
W 1 to W 4 (respectively of size 1 ⇥ c, 1 ⇥ c, l ⇥ c, l ⇥ c ), Top-n computes:
[Equation]
angle of the latent vector
[Equation]
selection of the top n points
[Equation] [Equation]
modulation coefﬁcients
[Equation]
modulation of R with cosine values
[Equation]
[Equation]
condition on the latent vector
[Equation]
[Equation]
The crux of the Top-n creation module consists in selecting the points that will be used to generate a set based on the value of the latent vector (Eqns (4, 5, 6)).
Unfortunately, the gradient of the argsort operation is 0 almost everywhere ( @ R [ s ] / @  = 0 ), and a mechanism has to be used in order to train the angles  and the MLP of Eq.
In Top-n, we modulate the representation of the selected points R [ s ] with the cosines c (Eq. 7).
This operation provides a path in the computational graph that does not go through the argsort , so that the gradients of  and a are not always 0.
For example:
[Equation]
Equations (4) to (8) build upon the Top-K pooling mechanism used by Gao & Ji (2019) for graph coarsening, but differ in several aspects.
First, Gao & Ji (2019) compute cosines between the angle a and the representations R .
This method may tend to select points that are similar.
On the contrary, we parametrize the angles and the representations independently, so that two points can have similar angles (in which case they will usually be selected together) but very diverse representations at the same time.
Second, Gao & Ji (2019) uses a multiplicative modulation ( X
=
X ref [ s ]  ˜ c ) in Eq.
(8) while we use a more expressive FiLM layer (Perez et al., 2018) that combines both additive and multiplicative modulation.
Finally, while previous works usually append the latent vector to each point, we propose to exploit the equivalence between summation and concatenation when a linear layer is applied.
This property can be written
[Equation]
for W = cat ( W 1 , W 2 ) .
Contrary to the concatenation (left-hand side), the sum (right-hand side) does not compute several times z T W 2 , which reduces the complexity of this layer from O ( n ( c + l ) c ) to O ( nc 2 + cl + nl ) .
Again, we combine the sum and multiplicative modulation in a FiLM layer to build a more expressive model in Eq.
Our algorithm retains the advantages of First-n creation, but replaces the arbitrary selection of the ﬁrst n points by a mechanism that can select the most relevant points for a given set.
Since it
[Table]
also decouples the number of points in the reference set from the number of points in the training examples, the size of the reference set becomes a hyperparameter of the model
Empirically, we observe a tradeoff: when using more points in the reference set, each point is updated less often which makes training slower; however, the model tends to avoid overﬁtting and generalize better.
Since First-n and Top-n use a ﬁxed set of reference points, one may wonder if they restrict the model expressivity.
We however show that it is not the case: used with a two-layer MLP, the First-n and
Top-n modules are universal approximators over sets.
Proposition 1 (Expressivity) .
For any set size n , maximal norm M and precision parameter ✏ , there is a 2-layer row-wise MLP f and reference points { x 1 , ..., x n } such that, for any set { y 1 , ..., y n } of points in R d with 8 i,
||
y i ||
 M there is a latent vector z of size nd ⇥ 1 that satisﬁes:
[Equation]
Top-n creation can be used in any GAN or autoencoder-based architecture as a replacement for other set creation methods.
It is however not suited to normalizing ﬂows, because it is based on a hard selection process (the value of the reference points that are not selected is not used) and is therefore non invertible.
E XPERIMENTS
We compare Top-n to other set and graph creation methods on several tasks: autoencoding point clouds on a set version of MNIST, generating realistic 3D structures on a synthetic molecule-like dataset, and generating varied valid molecules on the QM9 chemical dataset 4 .
All training curves are available in Appendix D.
5. S ET MNIST
We ﬁrst perform experiments on the SetMNIST benchmark, introduced in Zhang et al. (2019a).
The task consists in autoencoding point clouds that are built by thresholding the pixel values in MNIST images, adding noise on the locations and normalizing the coordinates.
Our goal is to show that Topn can favorably replace other set creation methods without having to tune the rest of the architecture extensively.
For this purpose, we use existing implementations of DSPN (Zhang et al., 2019a)
TSPN (Kosiorek et al., 2020) 5 , which are respectively a sort of diffusion model and a transformer-based autoencoder.
Experiment details can be found in Appendix C.
The results for both methods are very similar (Figure 1): the standard exchangeable i.i.d.
creation has poor performance and needs more epochs to be trained than First-n and Top-n. Since all sets in this dataset have the same cardinality (360 points), the better performance of Top-n cannot come from its better ability to process varying sizes, but rather shows that it is able to select the most relevant reference points for each set.
5. S YNTHETIC DATASET
Since SetMNIST does not allow to measure generation quality easily, we further benchmark the different set generators on a synthetic dataset for which the quality of the generated sets can be as-
[Table]
sessed.
This dataset is a simpliﬁed model of molecules in 3D, and retains some of its characteristics: i) atoms are never too close to each other ii) there is a bond between two atoms if and only if they are closer than a given distance (that depends on the atom types) iii) if formal charges are forbidden, each atom has a predeﬁned valency.
Each set is created via rejection sampling: points are drawn iteratively from a uniform distribution within a bounding box in R 3 .
The ﬁrst point is always accepted, and the next ones are accepted only if they satisfy a predeﬁned set of constraints: i) they are not closer to any other point than a given threshold min-distance .
ii) they are connected to the rest of the set, i.e., have at least one neighbor than a distance neighbor-distance iii) they do not have too many neighbors.
Our dataset is made of 2000 sets that have between 2 and 35 points, with 9 points per set on average.
It is a simpliﬁcation of real molecules in several aspects: there are only single bonds, angles between bonds are not constrained and the atom types are only deﬁned by the valency, which reﬂects the fact that atoms with the same valency tend to play a similar role and be more interchangeable.
The goal is to reconstruct the atom positions and generate new realistic sets.
During training, we measure the Wasserstein ( W 2 ) distance between the input and reconstructed sets.
At generation time, we compute the W 2 distance between the distribution of valencies in the dataset and in the generated set, as well as a diversity score to ensure that a method does not always generate the same set.
For that purpose we use the average W 2 distance between 1000 pairs of generated sets.
We also measure the same metrics in an extrapolation setting where sets have on average 10 more points.
Since the valency distribution may not be exactly the same on bigger sets, we also report the proportion of generated atoms whose valency is not present in the dataset (0, or more than 4).
In practice, we ﬁnd that the two measures are very correlated.
We train a VAE with different creation methods on this dataset.
Details about the model and the loss function can be found in Appendix C, as well as an ablation study on the number of points in the reference set.
The results are shown in Table 2.
We observe that the random i.i.d.
generator generalizes well but reconstructs the training sets very poorly, which reﬂects the fact that the model is hard to train due to the stochastic i.i.d.
generation.
The generated sets also lack diversity, which may reveal that the network tends to ignore the features that are drawn and rely on the layers biases.
On the contrary, the MLP and First-n generators seem to overﬁt the training data at the expense of generation quality.
Finally, Top-n is able to generate points that have the right valency and performance does not degrades when extrapolating to larger sets.
5. M OLECULAR GRAPH GENERATION
Finally, we evaluate Top-n on a graph generation task.
We train a graph VAE (detailed in Appendix
C) on QM9 molecules and check its ability to generate a wide range of valid molecules.
As a generation metric, we simply report the validity (i.e., the proportion of molecules that satisfy a set of basic chemical rules) and uniqueness (the proportion non-isomorphic graphs among valid ones) of the generated molecules.
We do not report novelty, as QM9 is an enumeration of all possible molecules up to 9 heavy atoms that satisfy a predeﬁned set of constraints (Ruddigkeit et al., 2012;
Ramakrishnan et al., 2014): in this setting, generating novel molecules is therefore not an indicator of good performance, but rather a sign that the distribution of the training data has not been properly captured.
Note that most recently proposed methods proposed on this task use a non-learned validity
[Table] [Table]
correction code to generate almost 100% of valid molecules, which obfuscates the real performance of the learned model.
We therefore only compare to works that do not correct validity: GraphVAE (Simonovsky & Komodakis, 2018), an extension of GraphVAE that incorporates reinforcement learning (Kwon et al., 2019), MolGAN (De Cao & Kipf, 2018) and GraphTransformerVAE (Mitton et al., 2021).
We observe that both MolGAN and the Graph VAE that incorporates reinforcement learning suffer from mode collapse: they can generate only a few molecules, but these molecules are often valid.
On the contrary, our method is able to generate a higher rate of valid and unique molecules.
We observe that the random i.i.d.
method is not able to obtain a good train loss (Appendix D), which reﬂects in the poor generation performance as well.
The three other set creation methods seem to perform similarly.
Our interpretation is that almost all molecules in QM9 have the same size ( heavy atoms, since hydrogens are not represented), which makes it less important to properly handle varying graph sizes as is proposed in Top-n.
We ﬁnally note that recursive methods such as Li et al.
(2018) can often generate higher rates of valid molecules.
The reason is that recursive methods can easily check at each step that the added edge will not break valency constraints.
They have therefore an unfair advantage over one-shot models which do not incorporate these checks.
C ONCLUSION
In this work, we strengthened the theoretical foundations of one-shot set and graph generation by proposing a deﬁnition of equivariant learning algorithms.
We showed in particular that, contrary to common belief, exchangeability is not a required property in GANs and VAE.
We then proposed
Top-n, a non-exchangeable model for one-shot set and graph creation.
Our method consistently achieves better performance on set generation, and is able to generate a wide range of valid molecular graphs.
Overall, Top-n is easy to train (as it is deterministic), it extrapolates well and is able to select the most relevant points for each set.
Our method can be incorporated in any GAN or VAE architecture and replace favorably other set creation methods.
We ﬁnally note that our experiments on graph generation and molecule generation in 3D feature only small graphs with 9 nodes on average.
This set size is in line with most literature on the topic (Meldgaard et al., 2021;
Satorras et al., 2021; Mitton et al., 2021), and we observe on the synthetic dataset that good reconstruction is signiﬁcantly harder to achieve on large sets.
This observation leads to a simple question that has not yet been investigated: can set and graph generation methods avoid the curse of dimensionality?
For MLP-based generators, the answer is probably no: as they learn vectors of size n max d that are then reshaped to matrices, standard universal approximation results will most likely yield a sample complexity that is exponential in n max .
Whether and under what circumstances other set generators can overcome this problem is an important question that conditions the success of one-shot models for larger sets and graphs.
Ethics statement Although our method can be used to generate any type of sets or graphs, it was primarily built and benchmarked with molecule generation applications in mind.
While learning based methods will probably not replace expert knowledge in the short term, they can already be used to generate hit molecules, i e., small and simple molecules that have promising activity on a predeﬁned target.
These molecules can then be reﬁned by computational chemists in order to optimise activity and develop new effective drugs.
We are therefore convinced that research on set and graph generation can have a direct positive impact on society.
Reproducibility statement All code is included in the supplementary materials.
Further details can be found in Appendix C