Original: D I
Revised: D I

Original: G RESS
Revised: G RESS

Original: : D ISCRETE D ENOISING DIFFUSION FOR GRAPH GENERATION
Revised: : D ISCRETE D ENOISING DIFFUSION FOR GRAPH GENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This work introduces DiGress, a discrete denoising diffusion model for generating graphs with categorical node and edge attributes.
Revised: This work introduces DiGress, a discrete denoising diffusion model for generating graphs with categorical node and edge attributes.

Original: Our model defines a diffusion process that progressively edits a graph with noise (adding or removing edges, changing the categories) and a graph transformer network that learns to revert this process.
Revised: Our model defines a diffusion process that progressively edits a graph with noise (adding or removing edges, changing the categories) and a graph transformer network that learns to revert this process.

Original: With these two ingredients in place, we reduce distribution learning over graphs to a simple sequence of classification tasks.
Revised: With these two ingredients in place, we reduce distribution learning over graphs to a simple sequence of classification tasks.

Original: We further improve sample quality by proposing a new Markovian noise model that preserves the marginal distribution of node and edge types during diffusion and by adding auxiliary graph-theoretic features derived from the noisy graph at each diffusion step.
Revised: We further improve sample quality by proposing a new Markovian noise model that preserves the marginal distribution of node and edge types during diffusion and by adding auxiliary graph-theoretic features derived from the noisy graph at each diffusion step.

Original: Finally, we propose a guidance procedure for conditioning the generation on graph-level features.
Revised: Finally, we propose a guidance procedure for conditioning the generation on graph-level features.

Original: Overall, DiGress achieves state-of-the-art performance on both molecular and non-molecular datasets, with up to 3x validity improvement on a dataset of planar graphs.
Revised: Overall, DiGress achieves state-of-the-art performance on both molecular and non-molecular datasets, with up to 3x validity improvement on a dataset of planar graphs.

Original: In particular, it is the first model that scales to the large GuacaMol dataset containing 1.3M drug-like molecules without using a molecule-specific representation such as SMILES or fragments.
Revised: In particular, it is the first model that scales to the large GuacaMol dataset containing 1.3M drug-like molecules without using a molecule-specific representation such as SMILES or fragments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Denoising diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) form a powerful class of generative models.
Revised: Denoising diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) form a powerful class of generative models.

Original: At a high-level, these models are trained to denoise diffusion trajectories, and produce new samples by sampling noise and denoising it recursively.
Revised: At a high-level, these models are trained to denoise diffusion trajectories, and produce new samples by sampling noise and denoising it recursively.

Original: Diffusion models have been used successfully in various settings, outperforming all other methods on image and video (Dhariwal & Nichol, 2021;
Revised: Diffusion models have been used successfully in various settings, outperforming all other methods on image and video (Dhariwal & Nichol, 2021;

Original: Ho et al., 2022).
Revised: Ho et al., 2022).

Original: These successes generate hope for building powerful models for graph generation, a task that has applications as diverse as molecule design (Liu et al., 2018), traffic modeling (Yu & Gu, 2019), and code completion (Brockschmidt et al., 2019).
Revised: These successes generate hope for building powerful models for graph generation, a task that has applications as diverse as molecule design (Liu et al., 2018), traffic modeling (Yu & Gu, 2019), and code completion (Brockschmidt et al., 2019).

Original: Generating graphs remains however challenging due to the unordered nature and sparsity properties of these structures.
Revised: Generating graphs remains however challenging due to the unordered nature and sparsity properties of these structures.

Original: Previous diffusion models for graphs proposed to embed the graphs in a continuous space and add
Revised: Previous diffusion models for graphs proposed to embed the graphs in a continuous space and add

Original: Gaussian noise to the node features and graph adjacency matrix (Niu et al., 2020; Jo et al., 2022).
Revised: Gaussian noise to the node features and graph adjacency matrix (Niu et al., 2020; Jo et al., 2022).

Original: This destroys the graph’s sparsity and creates fully connected noisy graphs for which structural information (such as connectivity or cycle counts) is not defined.
Revised: This destroys the graph’s sparsity and creates fully connected noisy graphs for which structural information (such as connectivity or cycle counts) is not defined.

Original: As a result, continuous diffusion can make it hard for the denoising network to capture the structural properties of the data.
Revised: As a result, continuous diffusion can make it hard for the denoising network to capture the structural properties of the data.

Original: In this work, we propose DiGress, a discrete denoising diffusion model for generating graphs with categorical node and edge attributes.
Revised: In this work, we propose DiGress, a discrete denoising diffusion model for generating graphs with categorical node and edge attributes.

Original: Our noise model is a Markov process that consists of successive graphs edits (edge addition or deletion, node or edge category edit) that can occur independently on each node or edge.
Revised: Our noise model is a Markov process that consists of successive graphs edits (edge addition or deletion, node or edge category edit) that can occur independently on each node or edge.

Original: In order to invert this diffusion process, we train a graph transformer network to predict the clean graph from a noisy input.
Revised: In order to invert this diffusion process, we train a graph transformer network to predict the clean graph from a noisy input.

Original: The resulting architecture is permutation equivariant and admits an evidence lower bound for likelihood estimation.
Revised: The resulting architecture is permutation equivariant and admits an evidence lower bound for likelihood estimation.

Original: We then propose algorithmic improvements to DiGress.
Revised: We then propose algorithmic improvements to DiGress.

Original: First, we show that performance is improved when using a noise model that preserves the marginal distribution of node and edge types during diffusion rather than uniform noise.
Revised: First, we show that performance is improved when using a noise model that preserves the marginal distribution of node and edge types during diffusion rather than uniform noise.

Original: Second, we augment the input of our denoising network with structural and spectral features at each diffusion step, as they are known to help overcome the limited representation power of graph neural networks (Xu et al., 2019).
Revised: Second, we augment the input of our denoising network with structural and spectral features at each diffusion step, as they are known to help overcome the limited representation power of graph neural networks (Xu et al., 2019).

Original: This is possible because, in contrast to Gaussian-based models, the noisy graphs of DiGress are typically not complete.
Revised: This is possible because, in contrast to Gaussian-based models, the noisy graphs of DiGress are typically not complete.

Original: Finally, we introduce a novel guidance procedure for conditioning graph generation on graph-level properties, which is a key requirement for many applications.
Revised: Finally, we introduce a novel guidance procedure for conditioning graph generation on graph-level properties, which is a key requirement for many applications.

Original: We present experimental evidence showing that DiGress is state-of-the-art on several benchmarks, as it produces a high rate of realistic graphs while maintaining high diversity and novelty.
Revised: We present experimental evidence showing that DiGress is state-of-the-art on several benchmarks, as it produces a high rate of realistic graphs while maintaining high diversity and novelty.

Original: Moreover, it matches the performance of autoregressive models on the large MOSES and GuacaMol molecular datasets, which were previously too large for one-shot models.
Revised: Moreover, it matches the performance of autoregressive models on the large MOSES and GuacaMol molecular datasets, which were previously too large for one-shot models.

Original: Overall, DiGress opens new perspectives for solving complex graph generation tasks with one-shot generative models.
Revised: Overall, DiGress opens new perspectives for solving complex graph generation tasks with one-shot generative models.

Original:  D IFFUSION MODELS
Revised:  D IFFUSION MODELS

Original: In this section, we first introduce the key ingredients of denoising diffusion models that do not depend on the data modality (image, text or graph).
Revised: In this section, we first introduce the key ingredients of denoising diffusion models that do not depend on the data modality (image, text or graph).

Original: Denoising diffusion models are defined by two components: a noise model and a denoising neural network.
Revised: Denoising diffusion models are defined by two components: a noise model and a denoising neural network.

Original: The noise model q takes as input a data point x and progressively corrupts it to create noisy data points ( z 1 , . .
Revised: The noise model q takes as input a data point x and progressively corrupts it to create noisy data points ( z 1 , . .

Original: , z T ) , which together form a trajectory of increasingly noisy data.
Revised: , z T ) , which together form a trajectory of increasingly noisy data.

Original: It is chosen to be Markov, so that q ( z 1 , . . .
Revised: It is chosen to be Markov, so that q ( z 1 , . . .

Original: , z T | x ) =
Revised: , z T | x ) =

Original: q ( z 1 | x ) (cid:81)
Revised: q ( z 1 | x ) (cid:81)

Original: Tt =2 q ( z t | z
Revised: Tt =2 q ( z t | z

Original: t − 1 ) .
Revised: t − 1 ) .

Original: The denoising network ϕ θ learns to invert these trajectories: it takes as input a noisy state z t and predicts ˆ z t − 1 =
Revised: The denoising network ϕ θ learns to invert these trajectories: it takes as input a noisy state z t and predicts ˆ z t − 1 =

Original: ϕ θ ( z t ) .
Revised: ϕ θ ( z t ) .

Original: To produce new samples, some noise z T is first sampled from a distribution q ∞ .
Revised: To produce new samples, some noise z T is first sampled from a distribution q ∞ .

Original: The network is then applied recursively in order to turn the noise into a realistic sample.
Revised: The network is then applied recursively in order to turn the noise into a realistic sample.

Original: While early diffusion models directly learnt to invert trajectories (Sohl-Dickstein et al., 2015), these models were difficult to train.
Revised: While early diffusion models directly learnt to invert trajectories (Sohl-Dickstein et al., 2015), these models were difficult to train.

Original: z t − 1 indeed depends on the sampling of both x and the diffusion trajectory, which creates some label noise during training.
Revised: z t − 1 indeed depends on the sampling of both x and the diffusion trajectory, which creates some label noise during training.

Original: Ho et al.
Revised: Ho et al.

Original: (2020) considerably improved performance by establishing a connection with score-based models (Song & Ermon, 2019), showing that when E p θ ( z t − 1 | z t , x ) is tractable, x can directly be used as the target of the denoising network.
Revised: (2020) considerably improved performance by establishing a connection with score-based models (Song & Ermon, 2019), showing that when E p θ ( z t − 1 | z t , x ) is tractable, x can directly be used as the target of the denoising network.

Original: Training ϕ
Revised: Training ϕ

Original: θ to predict x rather than z t − 1 is more efficient because z t − 1 does not need to be sampled, which removes a source of randomness.
Revised: θ to predict x rather than z t − 1 is more efficient because z t − 1 does not need to be sampled, which removes a source of randomness.

Original: Furthermore, the network can be trained in parallel on the different time steps and does not need to be applied recursively.
Revised: Furthermore, the network can be trained in parallel on the different time steps and does not need to be applied recursively.

Original: This confers a considerable stability advantage over other generative models like normalizing flows.
Revised: This confers a considerable stability advantage over other generative models like normalizing flows.

Original: In general, three properties are required to build an efficient diffusion model:
Revised: In general, three properties are required to build an efficient diffusion model:

Original: The distribution q ( z t | x ) should have a closed-form formula, to prevent applying noise recursively during training.
Revised: The distribution q ( z t | x ) should have a closed-form formula, to prevent applying noise recursively during training.

Original: The posterior q ( z t − 1 | z t , x ) should have a closed-form expression, so that x can be used as the target of the neural network.
Revised: The posterior q ( z t − 1 | z t , x ) should have a closed-form expression, so that x can be used as the target of the neural network.

Original: The limit distribution q ∞ =
Revised: The limit distribution q ∞ =

Original: lim T →∞ q ( z T | x ) should not depend on x , so that we can use it as a prior distribution for inference.
Revised: lim T →∞ q ( z T | x ) should not depend on x , so that we can use it as a prior distribution for inference.

Original: These properties are all satisfied when the noise is Gaussian.
Revised: These properties are all satisfied when the noise is Gaussian.

Original: When the task requires to model categorical data, Gaussian noise can still be used by embedding the data in a continuous space with a one-hot encoding of the categories (Niu et al., 2020; Jo et al., 2022).
Revised: When the task requires to model categorical data, Gaussian noise can still be used by embedding the data in a continuous space with a one-hot encoding of the categories (Niu et al., 2020; Jo et al., 2022).

Original: We develop in Appendix A a graph generation model based on this principle, and use it for ablation studies.
Revised: We develop in Appendix A a graph generation model based on this principle, and use it for ablation studies.

Original: However, Gaussian noise is a poor noise model for graphs as it destroys sparsity as well as graph theoretic notions such as connectivity.
Revised: However, Gaussian noise is a poor noise model for graphs as it destroys sparsity as well as graph theoretic notions such as connectivity.

Original: Discrete diffusion therefore seems more appropriate to graph generation tasks.
Revised: Discrete diffusion therefore seems more appropriate to graph generation tasks.

Original: Recent works have considered the discrete diffusion problem for text, image and audio data (Hoogeboom et al., 2021;
Revised: Recent works have considered the discrete diffusion problem for text, image and audio data (Hoogeboom et al., 2021;

Original: Johnson et al., 2021; Yang et al., 2022).
Revised: Johnson et al., 2021; Yang et al., 2022).

Original: We follow here the setting proposed by Austin et al.
Revised: We follow here the setting proposed by Austin et al.

Original: It considers a data point x that belongs to one of d classes and x ∈ R d its one-hot encoding.
Revised: It considers a data point x that belongs to one of d classes and x ∈ R d its one-hot encoding.

Original: The noise is now represented by transition matrices ( Q 1 , ..., Q T ) such that [ Q t ] ij represents the probability of jumping from state i to state j : q ( z t | z t − 1 ) =
Revised: The noise is now represented by transition matrices ( Q 1 , ..., Q T ) such that [ Q t ] ij represents the probability of jumping from state i to state j : q ( z t | z t − 1 ) =

Original: z t − 1 Q t .
Revised: z t − 1 Q t .

Original: As the process is Markovian, the transition matrix from x to z t reads ¯ Q t = Q 1 Q 2 ... Q t .
Revised: As the process is Markovian, the transition matrix from x to z t reads ¯ Q t = Q 1 Q 2 ... Q t .

Original: As long as ¯ Q t is precomputed or has a closed-form expression, the noisy states z t can be built from x using q ( z t | x ) = x ¯ Q t without having to apply noise recursively (Property 1).
Revised: As long as ¯ Q t is precomputed or has a closed-form expression, the noisy states z t can be built from x using q ( z t | x ) = x ¯ Q t without having to apply noise recursively (Property 1).

Original: The posterior distribution q ( z t − 1 | z t , x ) can also be computed in closed-form using Bayes rule (Property 2):
Revised: The posterior distribution q ( z t − 1 | z t , x ) can also be computed in closed-form using Bayes rule (Property 2):

Original: [Equation]
Revised: [Equation]

Original: where ⊙ denotes a pointwise product and Q ′ is the transpose of Q .
Revised: where ⊙ denotes a pointwise product and Q ′ is the transpose of Q .

Original: Finally, the limit distribution of the noise model depends on the transition model.
Revised: Finally, the limit distribution of the noise model depends on the transition model.

Original: The simplest and most common one is a uniform transition (Hoogeboom et al., 2021;
Revised: The simplest and most common one is a uniform transition (Hoogeboom et al., 2021;

Original: Austin et al., 2021;
Revised: Austin et al., 2021;

Original: Yang et al., 2022) parametrized by
Revised: Yang et al., 2022) parametrized by

Original: Q t = α t I + (1 − α t ) 1 d 1 ′ d /d with α t transitioning from 1 to 0.
Revised: Q t = α t I + (1 − α t ) 1 d 1 ′ d /d with α t transitioning from 1 to 0.

Original: When lim t →∞ α t = 0 , q ( z t | x ) converges to a uniform distribution independently of x (Property 3).
Revised: When lim t →∞ α t = 0 , q ( z t | x ) converges to a uniform distribution independently of x (Property 3).

Original: During inference, the predicted distribution is combined q
Revised: During inference, the predicted distribution is combined q

Original: G
Revised: G

Original: G | G order to compute
Revised: G | G order to compute

Original: G
Revised: G

Original: 1 | G t ) and sample a discrete G t − product of categorical
Revised: 1 | G t ) and sample a discrete G t − product of categorical

Original: The above framework satisfies all three properties in a setting that is inherently discrete.
Revised: The above framework satisfies all three properties in a setting that is inherently discrete.

Original: However, while it has been applied successfully to several data modalities, graphs have unique challenges that need to be considered: they have varying sizes, permutation equivariance properties, and to this date no known tractable universal approximator.
Revised: However, while it has been applied successfully to several data modalities, graphs have unique challenges that need to be considered: they have varying sizes, permutation equivariance properties, and to this date no known tractable universal approximator.

Original: In the next sections, we therefore propose a new discrete diffusion model that addresses the specific challenges of graph generation.
Revised: In the next sections, we therefore propose a new discrete diffusion model that addresses the specific challenges of graph generation.

Original:  D ISCRETE DENOISING DIFFUSION FOR GRAPH GENERATION (D I G RESS )
Revised:  D ISCRETE DENOISING DIFFUSION FOR GRAPH GENERATION (D I G RESS )

Original: In this section, we present the Di screte Graph Denoising Diffusion model (DiGress) (DiGress).
Revised: In this section, we present the Di screte Graph Denoising Diffusion model (DiGress) (DiGress).

Original: We consider graphs with categorical node and edge attributes.
Revised: We consider graphs with categorical node and edge attributes.

Original: We denote by X the space of node attributes, E the space of edge attributes, and write their respective cardinality a and b .
Revised: We denote by X the space of node attributes, E the space of edge attributes, and write their respective cardinality a and b .

Original: x i denotes the attribute of node i and x i ∈ R a its one-hot encoding.
Revised: x i denotes the attribute of node i and x i ∈ R a its one-hot encoding.

Original: These vectors are grouped in a matrix
Revised: These vectors are grouped in a matrix

Original: X ∈ R n × a where n is the number of nodes.
Revised: X ∈ R n × a where n is the number of nodes.

Original: Similarly, a tensor E ∈ R n × n × b groups the one-hot encoding e ij of each edge.
Revised: Similarly, a tensor E ∈ R n × n × b groups the one-hot encoding e ij of each edge.

Original: Importantly, we simply treat the absence of edge as a particular edge type .
Revised: Importantly, we simply treat the absence of edge as a particular edge type .

Original: A ′ denotes the matrix transpose of A , while A T is the value of A at time T .
Revised: A ′ denotes the matrix transpose of A , while A T is the value of A at time T .

Original: 3. D IFFUSION PROCESS AND INVERSE DENOISING ITERATIONS
Revised: 3. D IFFUSION PROCESS AND INVERSE DENOISING ITERATIONS

Original: We build on the discrete diffusion framework of Austin et al.
Revised: We build on the discrete diffusion framework of Austin et al.

Original: and model the diffusion process as a Markov process over X and E .
Revised: and model the diffusion process as a Markov process over X and E .

Original: Similarly to diffusion models for images, for which the noise is applied independently on each pixel, we diffuse separately on each node and edge feature.
Revised: Similarly to diffusion models for images, for which the noise is applied independently on each pixel, we diffuse separately on each node and edge feature.

Original: As a result, the state-space that we consider is not the one of graphs (which would be too large to build a transition matrix), but only the a node types and b edge types.
Revised: As a result, the state-space that we consider is not the one of graphs (which would be too large to build a transition matrix), but only the a node types and b edge types.

Original: For any node (resp. edge), the transition probabilities are defined by the matrices [ Q tX ] ij = q ( x t =
Revised: For any node (resp. edge), the transition probabilities are defined by the matrices [ Q tX ] ij = q ( x t =

Original: j | x t − 1 = i ) and [ Q tE ] ij = q ( e t = j | e t − 1 = i ) .
Revised: j | x t − 1 = i ) and [ Q tE ] ij = q ( e t = j | e t − 1 = i ) .

Original: Here, adding noise to form G t =
Revised: Here, adding noise to form G t =

Original: ( X t , E t ) means sampling each node and edge type from a categorical distribution defined by the following probabilities:
Revised: ( X t , E t ) means sampling each node and edge type from a categorical distribution defined by the following probabilities:

Original: [Equation]
Revised: [Equation]

Original: for ¯ Q tX =
Revised: for ¯ Q tX =

Original: Q tX ...
Revised: Q tX ...

Original: Q 1 X and ¯ Q tE =
Revised: Q 1 X and ¯ Q tE =

Original: Q tE ...
Revised: Q tE ...

Original: Q 1 E .
Revised: Q 1 E .

Original: When considering undirected graphs, we only apply noise to the upper-triangular part of E and then symmetrize the matrix.
Revised: When considering undirected graphs, we only apply noise to the upper-triangular part of E and then symmetrize the matrix.

Original: The second component of the diffusion model is the denoising neural network ϕ θ parametrized by θ .
Revised: The second component of the diffusion model is the denoising neural network ϕ θ parametrized by θ .

Original: It takes as input a noisy graph G t =
Revised: It takes as input a noisy graph G t =

Original: ( X t , E t ) and tries to predict the clean graph G , as illustrated in Figure 1.
Revised: ( X t , E t ) and tries to predict the clean graph G , as illustrated in Figure 1.

Original: To train it, we optimize the cross-entropy l between the predicted probabilities ˆ p G =
Revised: To train it, we optimize the cross-entropy l between the predicted probabilities ˆ p G =

Original: [Equation]
Revised: [Equation]

Original: ˆ p X , ˆ p E ) for each node and edge and the true graph G :
Revised: ˆ p X , ˆ p E ) for each node and edge and the true graph G :

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R + controls the relative importance of nodes and edges.
Revised: where λ ∈ R + controls the relative importance of nodes and edges.

Original: It is remarkable that, where architectures like VAEs solve a complex distribution learning problem that sometimes requires graph matching, our diffusion model simply solves classification tasks on each node and edge.
Revised: It is remarkable that, where architectures like VAEs solve a complex distribution learning problem that sometimes requires graph matching, our diffusion model simply solves classification tasks on each node and edge.

Original: Once the network is trained, it can be used to sample new graphs.
Revised: Once the network is trained, it can be used to sample new graphs.

Original: For this purpose, the reverse diffusion iterations p θ ( G t − 1 | G t ) need to be estimated from the network prediction ˆ p G .
Revised: For this purpose, the reverse diffusion iterations p θ ( G t − 1 | G t ) need to be estimated from the network prediction ˆ p G .

Original: We model this distribution as a product over nodes and edges:
Revised: We model this distribution as a product over nodes and edges:

Original: [Equation]
Revised: [Equation]

Original: To compute each term, we marginalize over the network predictions:
Revised: To compute each term, we marginalize over the network predictions:

Original: [Equation]
Revised: [Equation]

Original: where we choose
Revised: where we choose

Original: [Equation]
Revised: [Equation]

Original: if q
Revised: if q

Original: [Equation]
Revised: [Equation]

Original: x ti | x i = x ) > otherwise.
Revised: x ti | x i = x ) > otherwise.

Original: Similarly, we have p θ ( e t − 1 ij | e tij ) =
Revised: Similarly, we have p θ ( e t − 1 ij | e tij ) =

Original: (cid:80) e ∈E p θ ( e t − 1 ij | e ij = e
Revised: (cid:80) e ∈E p θ ( e t − 1 ij | e ij = e

Original: [Equation]
Revised: [Equation]

Original: e tij )
Revised: e tij )

Original: ˆ p
Revised: ˆ p

Original: Eij ( e ) .
Revised: Eij ( e ) .

Original: These distributions are used to sample a discrete G t − 1 that will be the input of the denoising network at the next time step.
Revised: These distributions are used to sample a discrete G t − 1 that will be the input of the denoising network at the next time step.

Original: These equations can also be used to compute an evidence lower bound on the likelihood, which allows for easier comparison between models.
Revised: These equations can also be used to compute an evidence lower bound on the likelihood, which allows for easier comparison between models.

Original: The computations are provided in Appendix C. We finally note that continuous features could additionally be considered, but they would require a separate definition of the noise.
Revised: The computations are provided in Appendix C. We finally note that continuous features could additionally be considered, but they would require a separate definition of the noise.

Original: 3. D ENOISING NETWORK PARAMETRIZATION
Revised: 3. D ENOISING NETWORK PARAMETRIZATION

Original: The denoising network takes as input a noisy graph G t =
Revised: The denoising network takes as input a noisy graph G t =

Original: ( X , E ) and outputs tensors
Revised: ( X , E ) and outputs tensors

Original: X ′ and E ′
Revised: X ′ and E ′

Original: (the predicted distribution over clean graphs).
Revised: (the predicted distribution over clean graphs).

Original: Internally, our layers also manipulate graph-level features y , as they are able to store information more efficiently.
Revised: Internally, our layers also manipulate graph-level features y , as they are able to store information more efficiently.

Original: Since attention mechanisms are a natural model for edge prediction, we chose to extend the graph transformer network proposed by
Revised: Since attention mechanisms are a natural model for edge prediction, we chose to extend the graph transformer network proposed by

Original: Dwivedi & Bresson (2021).
Revised: Dwivedi & Bresson (2021).

Original: Our model is schematized in Appendix B.1.
Revised: Our model is schematized in Appendix B.1.

Original: At a high-level, it first updates node features using self-attention.
Revised: At a high-level, it first updates node features using self-attention.

Original: Edge features and global features are incorporated to this self-attention mechanism using FiLM layers (Perez et al., 2018).
Revised: Edge features and global features are incorporated to this self-attention mechanism using FiLM layers (Perez et al., 2018).

Original: The edge features are then updated using the unnormalized attention scores, and the graph-level features using pooled node and edge features.
Revised: The edge features are then updated using the unnormalized attention scores, and the graph-level features using pooled node and edge features.

Original: As in other graph transformer networks, our model features residual connections and layer normalization.
Revised: As in other graph transformer networks, our model features residual connections and layer normalization.

Original: To incorporate time information into the model, we normalize the timestep to [0 , 1] and treat it as a global feature inside y .
Revised: To incorporate time information into the model, we normalize the timestep to [0 , 1] and treat it as a global feature inside y .

Original: Overall, our network has a memory and time complexity of Θ( n 2 ) per layer, due to the attention scores and the predictions for each edge.
Revised: Overall, our network has a memory and time complexity of Θ( n 2 ) per layer, due to the attention scores and the predictions for each edge.

Original: 3. E QUIVARIANCE PROPERTIES
Revised: 3. E QUIVARIANCE PROPERTIES

Original: Graphs are invariant to reorderings of their nodes, so that n !
Revised: Graphs are invariant to reorderings of their nodes, so that n !

Original: matrices can represent the same graph.
Revised: matrices can represent the same graph.

Original: For efficient learning, it is key to devise methods that do not require augmenting the data with random permutations, which implies that the loss and gradient updates should not change if the train data is permuted.
Revised: For efficient learning, it is key to devise methods that do not require augmenting the data with random permutations, which implies that the loss and gradient updates should not change if the train data is permuted.

Original: Two components are needed to achieve this property: a permutation equivariant architecture and a permutation invariant loss.
Revised: Two components are needed to achieve this property: a permutation equivariant architecture and a permutation invariant loss.

Original: We show that our method satisfies both properties.
Revised: We show that our method satisfies both properties.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: (Equivariance) DiGress is permutation equivariant.
Revised: (Equivariance) DiGress is permutation equivariant.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: (Invariant loss)
Revised: (Invariant loss)

Original: Any loss function (such as the cross-entropy loss of Eq.
Revised: Any loss function (such as the cross-entropy loss of Eq.

Original: (3)) that can be decomposed as (cid:80) i l X (ˆ p Xi , x i )
Revised: (3)) that can be decomposed as (cid:80) i l X (ˆ p Xi , x i )

Original: + (cid:80) i,j l E (ˆ p Eij , e ij ) for two functions l X and l E computed respectively on each node and each edge is permutation invariant.
Revised: + (cid:80) i,j l E (ˆ p Eij , e ij ) for two functions l X and l E computed respectively on each node and each edge is permutation invariant.

Original: Lemma 3.2 shows that our model does not require to match the predicted and target graphs, which would be difficult and costly.
Revised: Lemma 3.2 shows that our model does not require to match the predicted and target graphs, which would be difficult and costly.

Original: This is because the diffusion process keeps track of the identity of the nodes at each step – it can also be interpreted as a physical process where points are distinguishable.
Revised: This is because the diffusion process keeps track of the identity of the nodes at each step – it can also be interpreted as a physical process where points are distinguishable.

Original: Equivariance is however not a sufficient notion for likelihood computation: in general, the likelihood of a graph is the sum of the likelihood of all its permutations, which is intractable.
Revised: Equivariance is however not a sufficient notion for likelihood computation: in general, the likelihood of a graph is the sum of the likelihood of all its permutations, which is intractable.

Original: To avoid this computation, we can make sure that the generated distribution is exchangeable, i.e., that all permutations of generated graphs are equally likely (K¨ohler et al., 2020).
Revised: To avoid this computation, we can make sure that the generated distribution is exchangeable, i.e., that all permutations of generated graphs are equally likely (K¨ohler et al., 2020).

Original: Lemma 3.3.
Revised: Lemma 3.3.

Original: (Exchangeability)
Revised: (Exchangeability)

Original: DiGress yields exchangeable distributions, i.e., it generates graphs with node features X and adjacency matrix A that satisfy P ( X , A )
Revised: DiGress yields exchangeable distributions, i.e., it generates graphs with node features X and adjacency matrix A that satisfy P ( X , A )

Original: =
Revised: =

Original: P ( π T X , π T A π ) for any permutation π .
Revised: P ( π T X , π T A π ) for any permutation π .

Original:  I MPROVING D I G RESS WITH MARGINAL PROBABILITIES AND STRUCTURAL FEATURES 4. C HOICE OF THE NOISE MODEL
Revised:  I MPROVING D I G RESS WITH MARGINAL PROBABILITIES AND STRUCTURAL FEATURES 4. C HOICE OF THE NOISE MODEL

Original: The choice of the Markov transition matrices ( Q t ) t ≤ T defining the graph edit probabilities is arbitrary, and it is a priori not clear what noise model will lead to the best performance.
Revised: The choice of the Markov transition matrices ( Q t ) t ≤ T defining the graph edit probabilities is arbitrary, and it is a priori not clear what noise model will lead to the best performance.

Original: We have seen that the most common model is a uniform transition over the classes Q t = α t
Revised: We have seen that the most common model is a uniform transition over the classes Q t = α t

Original: I +(1 − α t )( 1 d 1 ′ d ) /d , and that it leads to limit distributions q X and q E that are uniform over categories.
Revised: I +(1 − α t )( 1 d 1 ′ d ) /d , and that it leads to limit distributions q X and q E that are uniform over categories.

Original: Graphs are however usually sparse, which means that the marginal distribution of edge types is far from uniform.
Revised: Graphs are however usually sparse, which means that the marginal distribution of edge types is far from uniform.

Original: Starting from uniform noise, we observe in Figure 2 that it takes many diffusion steps for the model to produce a sparse graph.
Revised: Starting from uniform noise, we observe in Figure 2 that it takes many diffusion steps for the model to produce a sparse graph.

Original: To improve upon this model, we propose the following hypothesis: training is easier when the prior distribution is close to the true data distribution .
Revised: To improve upon this model, we propose the following hypothesis: training is easier when the prior distribution is close to the true data distribution .

Original: This prior distribution cannot be chosen arbitrarily, as it needs to be permutation invariant to satisfy exchangeability (Lemma 3.3).
Revised: This prior distribution cannot be chosen arbitrarily, as it needs to be permutation invariant to satisfy exchangeability (Lemma 3.3).

Original: A natural model for this distribution is therefore a product (cid:81) i u × (cid:81)
Revised: A natural model for this distribution is therefore a product (cid:81) i u × (cid:81)

Original: i,j v of a single distribution u for all nodes and a single distribution v for all edges.
Revised: i,j v of a single distribution u for all nodes and a single distribution v for all edges.

Original: We propose the following result (proved in Appendix D) to guide the choice of u and v :
Revised: We propose the following result (proved in Appendix D) to guide the choice of u and v :

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: (Optimal prior distribution)
Revised: (Optimal prior distribution)

Original: Consider the class C
Revised: Consider the class C

Original: = { (cid:81) i u
Revised: = { (cid:81) i u

Original: × (cid:81) i,j v, ( u, v )
Revised: × (cid:81) i,j v, ( u, v )

Original: ∈ P ( X ) × P ( E ) } of distributions over graphs that factorize as the product of a single distribution u over X for the nodes and a single distribution v over E for the edges.
Revised: ∈ P ( X ) × P ( E ) } of distributions over graphs that factorize as the product of a single distribution u over X for the nodes and a single distribution v over E for the edges.

Original: Let P be an arbitrary distribution over graphs (seen as a tensor of order n + n 2 ) and m X , m E its marginal distributions of node and edge types.
Revised: Let P be an arbitrary distribution over graphs (seen as a tensor of order n + n 2 ) and m X , m E its marginal distributions of node and edge types.

Original: Then π G = (cid:81) i m X × (cid:81) i,j m E
Revised: Then π G = (cid:81) i m X × (cid:81) i,j m E

Original: is the orthogonal projection of P on C :
Revised: is the orthogonal projection of P on C :

Original: [Equation]
Revised: [Equation]

Original: This result means that to get a prior distribution q X × q E close to the true data distribution, we should define transition matrices such that ∀ i ≤ a, lim T →∞ ¯ Q TX 1 i
Revised: This result means that to get a prior distribution q X × q E close to the true data distribution, we should define transition matrices such that ∀ i ≤ a, lim T →∞ ¯ Q TX 1 i

Original: = m X (and similarly for edges).
Revised: = m X (and similarly for edges).

Original: To achieve this property, we propose to use
Revised: To achieve this property, we propose to use

Original: [Equation]
Revised: [Equation]

Original: With this model, the probability of jumping from a state i to a state j is proportional to the marginal probability of category j in the training set.
Revised: With this model, the probability of jumping from a state i to a state j is proportional to the marginal probability of category j in the training set.

Original: Since ( 1 m ′ ) 2 = 1 m ′ , we have ¯ Q t : ¯ Q t = ¯ α t
Revised: Since ( 1 m ′ ) 2 = 1 m ′ , we have ¯ Q t : ¯ Q t = ¯ α t

Original: I + ¯ β t 1 m ′ for ¯ α t
Revised: I + ¯ β t 1 m ′ for ¯ α t

Original: =
Revised: =

Original: (cid:81) tτ
Revised: (cid:81) tτ

Original: =
Revised: =

Original: α τ and ¯ β t = (cid:81) tτ
Revised: α τ and ¯ β t = (cid:81) tτ

Original: =1 (1 − α τ ) .
Revised: =1 (1 − α τ ) .

Original: We follow the popular cosine schedule ¯ α t = cos (0 .
Revised: We follow the popular cosine schedule ¯ α t = cos (0 .

Original: 5 π ( t/T + s ) / (1 + s )) 2 with a small s .
Revised: 5 π ( t/T + s ) / (1 + s )) 2 with a small s .

Original: We verify experimentally that these marginal transitions improves over uniform transitions (Appendix E).
Revised: We verify experimentally that these marginal transitions improves over uniform transitions (Appendix E).

Original: 4. S TRUCTURAL FEATURES AUGMENTATION
Revised: 4. S TRUCTURAL FEATURES AUGMENTATION

Original: Generative models for graphs inherit the limitations of graph neural networks, and in particular their limited representation power (Xu et al., 2019; Morris et al., 2019).
Revised: Generative models for graphs inherit the limitations of graph neural networks, and in particular their limited representation power (Xu et al., 2019; Morris et al., 2019).

Original: For example, standard message passing networks cannot detect simple substructures such as cycles (Chen et al., 2020),
Revised: For example, standard message passing networks cannot detect simple substructures such as cycles (Chen et al., 2020),

Original: Input: A graph G =
Revised: Input: A graph G =

Original: ( X , E )
Revised: ( X , E )

Original: Sample t ∼ U (1 , ..., T )
Revised: Sample t ∼ U (1 , ..., T )

Original: Sample G t ∼ X ¯ Q tX ×
Revised: Sample G t ∼ X ¯ Q tX ×

Original: E ¯ Q tE ▷ Sample a (discrete) noisy graph z ← f ( G t , t )
Revised: E ¯ Q tE ▷ Sample a (discrete) noisy graph z ← f ( G t , t )

Original: ▷ Structural and spectral features ˆ p X ,
Revised: ▷ Structural and spectral features ˆ p X ,

Original: ˆ p E ← ϕ θ ( G t , z ) ▷ Forward pass loss ← l CE (ˆ p X , X ) + λ l CE (ˆ p E , E ) optimizer .
Revised: ˆ p E ← ϕ θ ( G t , z ) ▷ Forward pass loss ← l CE (ˆ p X , X ) + λ l CE (ˆ p E , E ) optimizer .

Original: step( loss )
Revised: step( loss )

Original: Sample n from the training
Revised: Sample n from the training

Original: Sample G T ∼ q X ( n ) × q E ( n ) ▷ Random graph for t
Revised: Sample G T ∼ q X ( n ) × q E ( n ) ▷ Random graph for t

Original: = T to 1 do z ← f ( G t , t )
Revised: = T to 1 do z ← f ( G t , t )

Original: ▷ Structural and spectral features ˆ p X ,
Revised: ▷ Structural and spectral features ˆ p X ,

Original: ˆ p E ← ϕ θ ( G t , z ) ▷ Forward pass
Revised: ˆ p E ← ϕ θ ( G t , z ) ▷ Forward pass

Original: G t − 1 ∼ (cid:81) i p θ ( x t − 1 i | G t ) (cid:81) ij p θ ( e t − 1 ij | G t )
Revised: G t − 1 ∼ (cid:81) i p θ ( x t − 1 i | G t ) (cid:81) ij p θ ( e t − 1 ij | G t )

Original: ▷ Product of categorical distr.
Revised: ▷ Product of categorical distr.

Original: end return Gwhich casts doubt on their ability to correctly capture the properties of the data distribution.
Revised: end return Gwhich casts doubt on their ability to correctly capture the properties of the data distribution.

Original: While more powerful networks have been proposed (Maron et al., 2019; Vignac et al., 2020; Morris et al., 2022), they have a significantly higher computational complexity and are slow to train.
Revised: While more powerful networks have been proposed (Maron et al., 2019; Vignac et al., 2020; Morris et al., 2022), they have a significantly higher computational complexity and are slow to train.

Original: A cheaper strategy has been proposed in the graph network literature: instead of using more powerful graph networks, we can augment standard message passing networks with features that they cannot compute by themselves.
Revised: A cheaper strategy has been proposed in the graph network literature: instead of using more powerful graph networks, we can augment standard message passing networks with features that they cannot compute by themselves.

Original: Bouritsas et al.
Revised: Bouritsas et al.

Original: (2022) for example proposed to add counts of substructures of interest, and Beaini et al.
Revised: (2022) for example proposed to add counts of substructures of interest, and Beaini et al.

Original: (2021) proposed to add spectral features, as they are known to capture several global properties of graphs (Chung & Graham, 1997).
Revised: (2021) proposed to add spectral features, as they are known to capture several global properties of graphs (Chung & Graham, 1997).

Original: When the input to a neural network is continuous noise rather than a graph, as it is the case for most generative models, such auxiliary features cannot be added.
Revised: When the input to a neural network is continuous noise rather than a graph, as it is the case for most generative models, such auxiliary features cannot be added.

Original: However, DiGress operates on a discrete space and takes noisy graphs as input.
Revised: However, DiGress operates on a discrete space and takes noisy graphs as input.

Original: We can therefore compute various graph descriptors at each diffusion step , and input them to the network to help it denoise the graphs.
Revised: We can therefore compute various graph descriptors at each diffusion step , and input them to the network to help it denoise the graphs.

Original: This leads to Algorithms 1 and 2 for training DiGress and sampling from it.
Revised: This leads to Algorithms 1 and 2 for training DiGress and sampling from it.

Original: We refer to Appendix B.1 for a description of the features that we use in our experiments.
Revised: We refer to Appendix B.1 for a description of the features that we use in our experiments.

Original: We highlight the fact that these features empirically boost performance, but are not required to build a good model.
Revised: We highlight the fact that these features empirically boost performance, but are not required to build a good model.

Original: Depending on the graphs sizes, features such as cycle counts or spectral features (which are computed in O ( n 3 ) ) may or may not be used.
Revised: Depending on the graphs sizes, features such as cycle counts or spectral features (which are computed in O ( n 3 ) ) may or may not be used.

Original:  C ONDITIONAL GENERATION
Revised:  C ONDITIONAL GENERATION

Original: While good unconditional generation is a prerequisite, the ability to condition generation on several graph-level properties is key to many applications: for example, interesting molecules for drug design should be easy to synthesize and have high activity on some targets.
Revised: While good unconditional generation is a prerequisite, the ability to condition generation on several graph-level properties is key to many applications: for example, interesting molecules for drug design should be easy to synthesize and have high activity on some targets.

Original: One way to perform conditional generation is to train the denoising network using the target properties (Hoogeboom et al., 2022), but it requires to retrain the model when the conditioning properties changes.
Revised: One way to perform conditional generation is to train the denoising network using the target properties (Hoogeboom et al., 2022), but it requires to retrain the model when the conditioning properties changes.

Original: Instead, we build upon the classifier guidance algorithm (Sohl-Dickstein et al., 2015) and propose a new discrete guidance scheme.
Revised: Instead, we build upon the classifier guidance algorithm (Sohl-Dickstein et al., 2015) and propose a new discrete guidance scheme.

Original: Our method uses a regressor g η which is trained to predict target properties y of a clean graph G from a noisy version of G : g η ( G t )
Revised: Our method uses a regressor g η which is trained to predict target properties y of a clean graph G from a noisy version of G : g η ( G t )

Original: ≈ y
Revised: ≈ y

Original: ( G ) .
Revised: ( G ) .

Original: This regressor will guide the unconditional diffusion model ϕ θ : at each sampling step, after the unconditional distribution p θ ( G t − 1 | G t ) has been computed, the regressor is used to modulate the predicted distribution and push it towards graphs with the right properties.
Revised: This regressor will guide the unconditional diffusion model ϕ θ : at each sampling step, after the unconditional distribution p θ ( G t − 1 | G t ) has been computed, the regressor is used to modulate the predicted distribution and push it towards graphs with the right properties.

Original: Equations for the conditional denoising process are given by the following lemma:
Revised: Equations for the conditional denoising process are given by the following lemma:

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: (Conditional reverse noising process) (Dhariwal & Nichol, 2021)
Revised: (Conditional reverse noising process) (Dhariwal & Nichol, 2021)

Original: Denote ˙ q the noising process conditioned on y , q the unconditional noising process, and assume that ˙ q ( G t | G, y )
Revised: Denote ˙ q the noising process conditioned on y , q the unconditional noising process, and assume that ˙ q ( G t | G, y )

Original: = ˙
Revised: = ˙

Original: q ( G t | G ) .
Revised: q ( G t | G ) .

Original: Then we have ˙
Revised: Then we have ˙

Original: q ( G t − 1 | G t , y )
Revised: q ( G t − 1 | G t , y )

Original: ∝ q ( G t − 1 | G t ) ˙
Revised: ∝ q ( G t − 1 | G t ) ˙

Original: q ( y | G t − 1 ) .
Revised: q ( y | G t − 1 ) .

Original: Input: Diffusion model ϕ θ , graph properties regressor g , guidance scale λ , graph size n
Revised: Input: Diffusion model ϕ θ , graph properties regressor g , guidance scale λ , graph size n

Original: Sample G T q X
Revised: Sample G T q X

Original: ×
Revised: ×

Original: Random graph for to 1 do z
Revised: Random graph for to 1 do z

Original: , t ) ▷ Structural and spectral features ˆ p X ,
Revised: , t ) ▷ Structural and spectral features ˆ p X ,

Original: ˆ p E ← ϕ θ ( G t , z ) ▷ Forward pass ˆ y ← g η ( G t )
Revised: ˆ p E ← ϕ θ ( G t , z ) ▷ Forward pass ˆ y ← g η ( G t )

Original: ▷ Regressor model p η
Revised: ▷ Regressor model p η

Original: [Equation]
Revised: [Equation]

Original: ˆ y
Revised: ˆ y

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ⟩ ) ▷ Guidance distribution
Revised: ⟩ ) ▷ Guidance distribution

Original: Sample G t − 1 ∼ p θ ( G t − 1 | G t ) p
Revised: Sample G t − 1 ∼ p θ ( G t − 1 | G t ) p

Original: η
Revised: η

Original: (ˆ y | G t − 1 ) end
Revised: (ˆ y | G t − 1 ) end

Original: While we would like to estimate q ( G t − 1 | G t ) ˙
Revised: While we would like to estimate q ( G t − 1 | G t ) ˙

Original: q ( y | G t − 1 ) by p θ ( G t − 1 | G t )
Revised: q ( y | G t − 1 ) by p θ ( G t − 1 | G t )

Original: p η ( y | G t − 1 ) , p η cannot be evaluated for all possible values of G t − 1 .
Revised: p η ( y | G t − 1 ) , p η cannot be evaluated for all possible values of G t − 1 .

Original: To overcome this issue, we view G as a continuous tensor of order n + n 2 (so that ∇ G can be defined) and resort to a first-order approximation.
Revised: To overcome this issue, we view G as a continuous tensor of order n + n 2 (so that ∇ G can be defined) and resort to a first-order approximation.

Original: We have log ˙ q
Revised: We have log ˙ q

Original: ( y | G t − 1 )
Revised: ( y | G t − 1 )

Original: ≈ log ˙
Revised: ≈ log ˙

Original: [Equation]
Revised: [Equation]

Original: ( y | G t ) + ⟨∇ G log ˙
Revised: ( y | G t ) + ⟨∇ G log ˙

Original: [Equation]
Revised: [Equation]

Original: ( y | G t ) , G t − 1 − G t ⟩ ≈ c ( G t ) + (cid:88) 1 ≤ i ≤ n ⟨∇ x i log ˙
Revised: ( y | G t ) , G t − 1 − G t ⟩ ≈ c ( G t ) + (cid:88) 1 ≤ i ≤ n ⟨∇ x i log ˙

Original: [Equation]
Revised: [Equation]

Original: ( y | G t
Revised: ( y | G t

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: − 1 i ⟩ + (cid:88) 1 ≤
Revised: − 1 i ⟩ + (cid:88) 1 ≤

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ≤ n
Revised: ≤ n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: | G
Revised: | G

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ⟩ for a function c that does not depend on G t − 1 .
Revised: ⟩ for a function c that does not depend on G t − 1 .

Original: We make the additional assumption that ˙ q ( y | G t ) =
Revised: We make the additional assumption that ˙ q ( y | G t ) =

Original: N ( g ( G t ) , σ y I ) for an unknown function g , so that ∇ log ˙
Revised: N ( g ( G t ) , σ y I ) for an unknown function g , so that ∇ log ˙

Original: q η ( y | G t )
Revised: q η ( y | G t )

Original: ∝ −∇|| g ( G t ) − y ||
Revised: ∝ −∇|| g ( G t ) − y ||

Original: ∇ g is estimated by the regressor gradient ∇ g η , which can be computed in parallel for each node and edge.
Revised: ∇ g is estimated by the regressor gradient ∇ g η , which can be computed in parallel for each node and edge.

Original: The resulting procedure is presented in Algorithm 3.
Revised: The resulting procedure is presented in Algorithm 3.

Original: In addition to being conditioned on graph-level properties, our model can be used to extend an existing subgraph – a task called molecular scaffold extension in the drug discovery literature (Maziarz et al., 2022).
Revised: In addition to being conditioned on graph-level properties, our model can be used to extend an existing subgraph – a task called molecular scaffold extension in the drug discovery literature (Maziarz et al., 2022).

Original: In Appendix D.1, we explain how to do it and demonstrate it on a simple example.
Revised: In Appendix D.1, we explain how to do it and demonstrate it on a simple example.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Concurrently to our work, Haefeli et al.
Revised: Concurrently to our work, Haefeli et al.

Original: (2022) designed a discrete diffusion model for unattributed graphs using uniform transitions.
Revised: (2022) designed a discrete diffusion model for unattributed graphs using uniform transitions.

Original: Discrete diffusion models have also been proposed for text, images, audio data and attributed point clouds (Austin et al., 2021;
Revised: Discrete diffusion models have also been proposed for text, images, audio data and attributed point clouds (Austin et al., 2021;

Original: Yang et al., 2022; Luo et al., 2022).
Revised: Yang et al., 2022; Luo et al., 2022).

Original: While DiGress is the first discrete diffusion model for graphs, two continuous models have been proposed for the same task:
Revised: While DiGress is the first discrete diffusion model for graphs, two continuous models have been proposed for the same task:

Original: Niu et al.
Revised: Niu et al.

Original: (2020) generate adjacency matrices by thresholding a continuous value to indicate edges, and Jo et al.
Revised: (2020) generate adjacency matrices by thresholding a continuous value to indicate edges, and Jo et al.

Original: (2022) extend this model to handle node and edge attributes.
Revised: (2022) extend this model to handle node and edge attributes.

Original: Trippe et al.
Revised: Trippe et al.

Original: (2022), Hoogeboom et al.
Revised: (2022), Hoogeboom et al.

Original: (2022) and Wu et al.
Revised: (2022) and Wu et al.

Original: (2022) define diffusion models for molecule generation in 3d.
Revised: (2022) define diffusion models for molecule generation in 3d.

Original: These models actually solve a point cloud generation task, as they generate the position of the atoms but not the presence of bonds: they therefore need to be trained on conformer data, while our model operates on graphs.
Revised: These models actually solve a point cloud generation task, as they generate the position of the atoms but not the presence of bonds: they therefore need to be trained on conformer data, while our model operates on graphs.

Original: On the contrary, Xu et al.
Revised: On the contrary, Xu et al.

Original: and Jing et al.
Revised: and Jing et al.

Original: (2022) define diffusion model for conformation generation – they input a graph structure and output atomic coordinates.
Revised: (2022) define diffusion model for conformation generation – they input a graph structure and output atomic coordinates.

Original: Such models could complement DiGress to generate molecules in 3d.
Revised: Such models could complement DiGress to generate molecules in 3d.

Original: Apart from diffusion models, there has recently been a lot of interest in non-autoregressive graph generation using VAEs, GANs or normalizing flows (Zhu et al., 2022).
Revised: Apart from diffusion models, there has recently been a lot of interest in non-autoregressive graph generation using VAEs, GANs or normalizing flows (Zhu et al., 2022).

Original: Three such models (Madhawa et al., 2019; Lippe & Gavves, 2021; Luo et al., 2021) operate over discrete data using categorical normalizing flows.
Revised: Three such models (Madhawa et al., 2019; Lippe & Gavves, 2021; Luo et al., 2021) operate over discrete data using categorical normalizing flows.

Original: On molecular tasks, these methods however do not match the performance of autoregressive (Liu et al., 2018;
Revised: On molecular tasks, these methods however do not match the performance of autoregressive (Liu et al., 2018;

Original: Liao et al., 2019; Mercado et al., 2021) and motifs-based models (Jin et al., 2020; Maziarz et al., 2022), which are able to incorporate much more domain knowledge.
Revised: Liao et al., 2019; Mercado et al., 2021) and motifs-based models (Jin et al., 2020; Maziarz et al., 2022), which are able to incorporate much more domain knowledge.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We show on both molecular and non-molecular benchmarks that DiGress significantly improves upon existing one-shot graph generation methods .In particular, it can generate high quality drugsized molecules where most previous methods were limited to very small atomic graphs.
Revised: We show on both molecular and non-molecular benchmarks that DiGress significantly improves upon existing one-shot graph generation methods .In particular, it can generate high quality drugsized molecules where most previous methods were limited to very small atomic graphs.

Original: We
Revised: We

Original: [Table]
Revised: [Table]

Original: compare its performance against Set2GraphVAE (Vignac & Frossard, 2021)
Revised: compare its performance against Set2GraphVAE (Vignac & Frossard, 2021)

Original: [Table]
Revised: [Table]

Original: SPECTRE (Martinkus et al., 2022), GraphNVP (Madhawa et al., 2019)
Revised: SPECTRE (Martinkus et al., 2022), GraphNVP (Madhawa et al., 2019)

Original: [Table]
Revised: [Table]

Original: GDSS (Jo et al., 2022), GraphRNN (You et al., 2018), GRAN (Liao et al., 2019), JT-VAE (Jin et al., 2018), NAGVAE (Kwon et al., 2020) and GraphINVENT (Mercado et al., 2021).
Revised: GDSS (Jo et al., 2022), GraphRNN (You et al., 2018), GRAN (Liao et al., 2019), JT-VAE (Jin et al., 2018), NAGVAE (Kwon et al., 2020) and GraphINVENT (Mercado et al., 2021).

Original: We also build a Continuous Graph denoising model (ConGress) that has the same denoising network as DiGress but inside a continuous framework (Appendix A).
Revised: We also build a Continuous Graph denoising model (ConGress) that has the same denoising network as DiGress but inside a continuous framework (Appendix A).

Original: Our results are presented without validity correction or molecule optimization.
Revised: Our results are presented without validity correction or molecule optimization.

Original: 7. G ENERAL GRAPH GENERATION
Revised: 7. G ENERAL GRAPH GENERATION

Original: [Table]
Revised: [Table]

Original: ConGress 34.
Revised: ConGress 34.

Original: DiGress 1 .1 .1.
Revised: DiGress 1 .1 .1.

Original: Planar graphs
Revised: Planar graphs

Original: GraphRNN 24.
Revised: GraphRNN 24.

Original: GRAN 3.
Revised: GRAN 3.

Original: SPECTRE 2.
Revised: SPECTRE 2.

Original: ConGress 23.
Revised: ConGress 23.

Original: DiGress 1 .1 .

Revised: DiGress 1 .1 .

Original: We first consider the benchmark proposed in Martinkus et al.
Revised: We first consider the benchmark proposed in Martinkus et al.

Original: We use two datasets made of 200 graphs: one drawn from the stochastic block model (with up to 200 nodes per graphs), and another dataset of planar graphs (64 nodes per graph).
Revised: We use two datasets made of 200 graphs: one drawn from the stochastic block model (with up to 200 nodes per graphs), and another dataset of planar graphs (64 nodes per graph).

Original: We evaluate the ability of the models to correctly model various properties of these graphs.
Revised: We evaluate the ability of the models to correctly model various properties of these graphs.

Original: In particular, we measure if the generated graphs are statistically distinguishable from the SBM model, or if they are planar and connected.
Revised: In particular, we measure if the generated graphs are statistically distinguishable from the SBM model, or if they are planar and connected.

Original: We refer to Appendix E for a description of the metrics.
Revised: We refer to Appendix E for a description of the metrics.

Original: In Table 1, we observe that DiGress is able to capture the data distribution very well, with spectacular improvements over baselines on planar graphs.
Revised: In Table 1, we observe that DiGress is able to capture the data distribution very well, with spectacular improvements over baselines on planar graphs.

Original: This contrasts with our continuous model, which does not perform well on these relatively large graphs.
Revised: This contrasts with our continuous model, which does not perform well on these relatively large graphs.

Original: 7. S MALL MOLECULE GENERATION
Revised: 7. S MALL MOLECULE GENERATION

Original: We then evaluate our model on the standard QM9 dataset (Wu et al., 2018) that contains molecules with up to 9 heavy atoms.
Revised: We then evaluate our model on the standard QM9 dataset (Wu et al., 2018) that contains molecules with up to 9 heavy atoms.

Original: We keep 100k molecules for training, 20k for validation and 13k for evaluating likelihood on a test set.
Revised: We keep 100k molecules for training, 20k for validation and 13k for evaluating likelihood on a test set.

Original: We report the negative log-likelihood of our model, validity (measured by RDKit sanitization) and uniqueness over 10k molecules.
Revised: We report the negative log-likelihood of our model, validity (measured by RDKit sanitization) and uniqueness over 10k molecules.

Original: Novelty results are discussed in Appendix E. 95% confidence intervals are reported on five runs.
Revised: Novelty results are discussed in Appendix E. 95% confidence intervals are reported on five runs.

Original: Results are presented in Figure 2.
Revised: Results are presented in Figure 2.

Original: Since ConGress and DiGress both obtain close to perfect metrics on this dataset, we also perform in Appendix E an ablation study on a more difficult version of QM9 where hydrogens are explicitly modeled.
Revised: Since ConGress and DiGress both obtain close to perfect metrics on this dataset, we also perform in Appendix E an ablation study on a more difficult version of QM9 where hydrogens are explicitly modeled.

Original: It shows that the discrete framework is beneficial and that marginal transitions and auxiliary features further boost performance.
Revised: It shows that the discrete framework is beneficial and that marginal transitions and auxiliary features further boost performance.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 7. C ONDITIONAL GENERATION
Revised: 7. C ONDITIONAL GENERATION

Original: [Table]
Revised: [Table]

Original: To measure the ability of DiGress to condition the generation on graph-level properties, we propose a conditional generation setting on
Revised: To measure the ability of DiGress to condition the generation on graph-level properties, we propose a conditional generation setting on

Original: QM9.
Revised: QM9.

Original: We first sample 100 molecules from the test set and retrieve their dipole moment µ and the highest occupied molecular orbit (HOMO).
Revised: We first sample 100 molecules from the test set and retrieve their dipole moment µ and the highest occupied molecular orbit (HOMO).

Original: The pairs ( µ, HOMO ) constitute the conditioning vector that we use to generate 10 molecules.
Revised: The pairs ( µ, HOMO ) constitute the conditioning vector that we use to generate 10 molecules.

Original: To evaluate the ability of a model to condition correctly, we need to estimate the properties of the generated samples.
Revised: To evaluate the ability of a model to condition correctly, we need to estimate the properties of the generated samples.

Original: To do so, we first use RdKit (Landrum et al., 2006) to produce conformers of the generated graphs, and then use Psi4 (Smith et al., 2020) to estimate the values of µ and HOMO.
Revised: To do so, we first use RdKit (Landrum et al., 2006) to produce conformers of the generated graphs, and then use Psi4 (Smith et al., 2020) to estimate the values of µ and HOMO.

Original: We report the mean absolute error between the targets and the estimated values for the generated molecules (Figure 4).
Revised: We report the mean absolute error between the targets and the estimated values for the generated molecules (Figure 4).

Original: 7. M OLECULE GENERATION AT SCALE
Revised: 7. M OLECULE GENERATION AT SCALE

Original: We finally evaluate our model on two much more challenging datasets made of more than a million molecules: MOSES (Polykovskiy et al., 2020), which contains small drug-like molecules, and GuacaMol (Brown et al., 2019), which contains larger molecules.
Revised: We finally evaluate our model on two much more challenging datasets made of more than a million molecules: MOSES (Polykovskiy et al., 2020), which contains small drug-like molecules, and GuacaMol (Brown et al., 2019), which contains larger molecules.

Original: DiGress is to our knowledge the first one-shot generative model that is not based on molecular fragments and that scales to datasets of this size.
Revised: DiGress is to our knowledge the first one-shot generative model that is not based on molecular fragments and that scales to datasets of this size.

Original: The metrics used  are presented in App.
Revised: The metrics used as well as additional experiments are presented in App.

Original: E.
Revised: E.

Original: For MOSES, the reported scores for FCD, SNN, and Scaffold similarity are computed on the dataset made of separate scaffolds, which measures the ability of the networks to predict new ring structures.
Revised: For MOSES, the reported scores for FCD, SNN, and Scaffold similarity are computed on the dataset made of separate scaffolds, which measures the ability of the networks to predict new ring structures.

Original: Results are presented in Tables 3 and 4: they show that DiGress does not yet match the performance of SMILES and fragment-based methods, but performs on par with GraphInvent, thus bridging the important gap between one-shot methods and autoregressive models that previously prevailed.
Revised: Results are presented in Tables 3 and 4: they show that DiGress does not yet match the performance of SMILES and fragment-based methods, but performs on par with GraphInvent, thus bridging the important gap between one-shot methods and autoregressive models that previously prevailed.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we proposed DiGress, a new discrete denoising diffusion model for graph generation.
Revised: In this work, we proposed DiGress, a new discrete denoising diffusion model for graph generation.

Original: Our model learns to progressively edit a random graph in order to turn it into a realistic graph.
Revised: Our model learns to progressively edit a random graph in order to turn it into a realistic graph.

Original: It can be conditioned both on graph-level properties and on predefined subgraphs.
Revised: It can be conditioned both on graph-level properties and on predefined subgraphs.

Original: DiGress outperforms existing one-shot generation methods and is able to scale to larger datasets, reaching the performance of autogressive models on molecule generation.
Revised: DiGress outperforms existing one-shot generation methods and is able to scale to larger datasets, reaching the performance of autogressive models on molecule generation.

Original: A C ONTINUOUS G RAPH D ENOISING DIFFUSION MODEL (C ON G RESS )
Revised: A C ONTINUOUS G RAPH D ENOISING DIFFUSION MODEL (C ON G RESS )

Original: In this section we present a diffusion model for graphs that uses Gaussian noise rather than a discrete diffusion process.
Revised: In this section we present a diffusion model for graphs that uses Gaussian noise rather than a discrete diffusion process.

Original: Its denoising network is the same as the one of our discrete model.
Revised: Its denoising network is the same as the one of our discrete model.

Original: Our goal is to show that the better performance obtained with DiGress is not only due to the neural network design but also to the discrete process itself
Revised: Our goal is to show that the better performance obtained with DiGress is not only due to the neural network design but also to the discrete process itself.

Original: A. D IFFUSION PROCESS
Revised: A. D IFFUSION PROCESS

Original: Consider a graph G =
Revised: Consider a graph G =

Original: ( X , E ) .
Revised: ( X , E ) .

Original: Similarly to the discrete diffusion model this diffusion process adds noise independently on each node and each edge but this time the noise considered is Gaussian:
Revised: Similarly to the discrete diffusion model, this diffusion process adds noise independently on each node and each edge, but this time the noise considered is Gaussian:

Original: [Equation]
Revised: [Equation]

Original: N ( α t | t − 1 X t − 1 , ( σ t | t − 1 ) 2 I ) and q ( E t | E t − 1 )

Original: =

Original: N

Original: [Equation]

Original: α t |

Original: [Equation]

Original: This process can equivalently be written:
Revised: This process can equivalently be written:

Original: [Equation]
Revised: [Equation]

Original: where α t | t − 1 =
Revised: where α t | t − 1 =

Original: α t /α t − 1 and ( σ t | t − 1 ) 2 = ( σ t ) 2 − ( α t | t − 1 ) 2 ( σ t − 1 ) 2 .
Revised: α t /α t − 1 and ( σ t | t − 1 ) 2 = ( σ t ) 2 − ( α t | t − 1 ) 2 ( σ t − 1 ) 2 .

Original: The variance is chosen as ( σ t ) 2 = 1 − ( α t ) 2 in order to obtain a variance-preserving process (Kingma et al., 2021).
Revised: The variance is chosen as ( σ t ) 2 = 1 − ( α t ) 2 in order to obtain a variance-preserving process (Kingma et al., 2021).

Original: Similarly to DiGress, when we consider undirected graphs, we only apply the noise on the upper-triangular part of E without the main diagonal, and then symmetrize the matrix.
Revised: Similarly to DiGress, when we consider undirected graphs, we only apply the noise on the upper-triangular part of E without the main diagonal, and then symmetrize the matrix.

Original: The true denoising process can be computed in closed-form:
Revised: The true denoising process can be computed in closed-form:

Original: [Equation]
Revised: [Equation]

Original: and similarlyfor E
Revised: and similarly

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  with
Revised: E ) , (9) with

Original: [Equation]
Revised: [Equation]

Original: As commonly done for Gaussian diffusion models, we train the denoising network to predict the noise components ˆ ϵ X , ˆ ϵ E instead of ˆ X
Revised: As commonly done for Gaussian diffusion models, we train the denoising network to predict the noise components ˆ ϵ X , ˆ ϵ E instead of ˆ X

Original: and ˆ E themselves (Ho et al., 2020).
Revised: and ˆ E themselves (Ho et al., 2020).

Original: Both relate as follows:
Revised: Both relate as follows:

Original: [Equation]
Revised: [Equation]

Original:  To optimize the network, we minimize the mean squared error between the predicted noise and the true noise, which results in Algorithm 4 for training ConGress.
Revised: and [Equation] To optimize the network, we minimize the mean squared error between the predicted noise and the true noise, which results in Algorithm 4 for training ConGress.

Original: Sampling is done similarly tostandard Gaussian diffusion models, except for the last step: since continuous valued features are obtained, they must be mapped back to categorical values in order to obtain a discrete graph. For this purpose, we then take the argmax of X 0 , E 0 across node and edge types (Algorithm 5).
Revised: Sampling is done similarly to

Original: Input: A graph G =
Revised: Input: A graph G =

Original: ( X , E )
Revised: ( X , E )

Original: Sample t ∼ U (1 , ..., T )
Revised: Sample t ∼ U (1 , ..., T )

Original: Sample ϵ
Revised: Sample ϵ

Original: X ∼ N (0 , I n )
Revised: X ∼ N (0 , I n )

Original: Sample ϵ E ∼ N (0 , I n ( n − 1) / 2 ) and symmetrize if needed z t ← α t ( X , E ) + σ t ( ϵ X , ϵ E )
Revised: Sample ϵ E ∼ N (0 , I n ( n − 1) / 2 ) and symmetrize if needed z t ← α t ( X , E ) + σ t ( ϵ X , ϵ E )

Original: ▷ Add noise
Revised: ▷ Add noise

Original: Minimize || ( ϵ X , ϵ E ) − ϕ θ ( z t , t )
Revised: Minimize || ( ϵ X , ϵ E ) − ϕ θ ( z t , t )

Original: ||
Revised: ||

Original: Sample n from the training data distribution
Revised: Sample n from the training data distribution

Original: Sample ϵ
Revised: Sample ϵ

Original: X ∼ N (0 , I n )
Revised: X ∼ N (0 , I n )

Original: Sample ϵ E ∼ N (0 , I n ( n − 1) / 2 ) and symmetrize if needed z T ← ( ϵ X , ϵ E ) for t =
Revised: Sample ϵ E ∼ N (0 , I n ( n − 1) / 2 ) and symmetrize if needed z T ← ( ϵ X , ϵ E ) for t =

Original: T to 1 do
Revised: T to 1 do

Original: Sample ϵ
Revised: Sample ϵ

Original: X ∼ N (0 , I n )
Revised: X ∼ N (0 , I n )

Original: Sample and symmetrize ϵ E ∼ N (0 , I n ( n − 1) / 2 )
Revised: Sample and symmetrize ϵ E ∼ N (0 , I n ( n − 1) / 2 )

Original: z t − 1 ← 1 α t | t − 1 z t − σ 2 t | t − 1 α t | t − 1 σ t
Revised: z t − 1 ← 1 α t | t − 1 z t − σ 2 t | t − 1 α t | t − 1 σ t

Original: ϕ
Revised: ϕ

Original: θ
Revised: θ

Original: ( z t , t ) + σ t → t − 1 ( ϵ X , ϵ E )
Revised: ( z t , t ) + σ t → t − 1 ( ϵ X , ϵ E )

Original: ▷ Reverse iterations end return argmax( X 0 ) , argmax( E 0 ) 
Revised: ▷ Reverse iterations end return argmax( X 0 ) , argmax( E 0 ) standard Gaussian diffusion models, except for the last step: since continuous valued features are obtained, they must be mapped back to categorical values in order to obtain a discrete graph. For this purpose, we then take the argmax of X 0 , E 0 across node and edge types (Algorithm 5).

Original: Overall, ConGress is very close to the GDSS model proposed in Jo et al.
Revised: Overall, ConGress is very close to the GDSS model proposed in Jo et al.

Original: (2022), as it is also a
Revised: (2022), as it is also a

Original: Gaussian-based diffusion model for graphs.
Revised: Gaussian-based diffusion model for graphs.

Original: An important difference is that we define a diffusion process that is independent for each node and edge, while GDSS uses a more complex noise model that does not factorize.
Revised: An important difference is that we define a diffusion process that is independent for each node and edge, while GDSS uses a more complex noise model that does not factorize.

Original: We observe empirically that a simple noise model does not hurt performance, since ConGress outperforms GDSS on QM9 (Table 2).
Revised: We observe empirically that a simple noise model does not hurt performance, since ConGress outperforms GDSS on QM9 (Table 2).

Original: B N EURAL NETWORK PARAMETRIZATION B. G RAPHTRANSFORMER NETWORK
Revised: B N EURAL NETWORK PARAMETRIZATION B. G RAPH

Revised: [Figure]

Original: The parametrization of our denoising network is presented in Figure 5.
Revised: The parametrization of our denoising network is presented in Figure 5.

Original: It takes as input a noisy graph ( X , E ) and predicts a distribution over the clean graphs.
Revised: It takes as input a noisy graph ( X , E ) and predicts a distribution over the clean graphs.

Original: We compute structural and spectral features in order to improve the network expressivity.
Revised: We compute structural and spectral features in order to improve the network expressivity.

Original: Internally, each layer manipulates nodes features X , edge features E but also graph level features y .
Revised: Internally, each layer manipulates nodes features X , edge features E but also graph level features y .

Original: Each graph transformer layer is made of a graph attention module (presented in Figure 6), as well as a fully-connected layers and layer normalization.
Revised: Each graph transformer layer is made of a graph attention module (presented in Figure 6), as well as a fully-connected layers and layer normalization.

Original: B. A UXILIARY S TRUCTURAL AND SPECTRAL FEATURES
Revised: B. A UXILIARY S TRUCTURAL AND SPECTRAL FEATURES

Original: The structural features that we use can be divided in two types: graphtheoretic (cycles and spectral features) and domain specific (molecular features).
Revised: The structural features that we use can be divided in two types: graphtheoretic (cycles and spectral features) and domain specific (molecular features).

Original: Cycles Since message-passing neural networks are unable to detect cycles (Chen et al., 2020), we add cycle counts to our model.
Revised: Cycles Since message-passing neural networks are unable to detect cycles (Chen et al., 2020), we add cycle counts to our model.

Original: Because computing traversals would be impractical on GPUs (all the more as these features are recomputed at every diffusion step), we use formulas for cycles up to size 6.
Revised: Because computing traversals would be impractical on GPUs (all the more as these features are recomputed at every diffusion step), we use formulas for cycles up to size 6.

Original: We build node features (how many k − cycles does this node belong to?)
Revised: We build node features (how many k − cycles does this node belong to?)

Original: for up to 5-cycles, and graph-level features (how many k − cycles does thisgraph contain?) for up to k = 6 . We use the following formulas, where d denotes the vector containing node degrees and || . || F is Frobenius norm:
Revised: for up to 5-cycles, and graph-level features (how many k − cycles does this

Original: [Equation]
Revised: [Equation]

Original: Spectral features We also add the option to incorporate spectral features to the model.
Revised: Spectral features We also add the option to incorporate spectral features to the model.

Original: While this requires a O ( n 3 ) eigendecomposition, we find that it is not a limiting factor for the graphs that we use in our experiments (that have up to 200 nodes).
Revised: While this requires a O ( n 3 ) eigendecomposition, we find that it is not a limiting factor for the graphs that we use in our experiments (that have up to 200 nodes).

Original: We first compute some graph-level features that relate to the eigenvalues of the graph Laplacian: the number of connected components (given by the multiplicity of eigenvalue 0), as well as the 5 first nonzero eigenvalues.
Revised: We first compute some graph-level features that relate to the eigenvalues of the graph Laplacian: the number of connected components (given by the multiplicity of eigenvalue 0), as well as the 5 first nonzero eigenvalues.

Original: We then add node-level features relative to the graph eigenvectors: an estimation of the biggest connected component (using the eigenvectors associated to eigenvalue 0), as well as the two first eigenvectors associated to non zero eigenvalues.
Revised: We then add node-level features relative to the graph eigenvectors: an estimation of the biggest connected component (using the eigenvectors associated to eigenvalue 0), as well as the two first eigenvectors associated to non zero eigenvalues.

Original: Molecular features On molecular datasets, we also incorporate the current valency of each atom and the current molecular weight of the full molecule.
Revised: Molecular features On molecular datasets, we also incorporate the current valency of each atom and the current molecular weight of the full molecule.

Original: C L IKELIHOOD COMPUTATION
Revised: C L IKELIHOOD COMPUTATION

Original: The graphical model associated to our problem is presented in figure 7: the graph size is sampled from the training distribution and kept constant during diffusion.
Revised: The graphical model associated to our problem is presented in figure 7: the graph size is sampled from the training distribution and kept constant during diffusion.

Original: One can notice the similarity between this graphical model and hierarchical variational autoencoders (VAEs): diffusion models can in fact be interpreted as a particular instance of VAE where the encoder (i.e., the diffusion process) is fixed.
Revised: One can notice the similarity between this graphical model and hierarchical variational autoencoders (VAEs): diffusion models can in fact be interpreted as a particular instance of VAE where the encoder (i.e., the diffusion process) is fixed.

Original: The likelihood of a data point x under the model writes:
Revised: The likelihood of a data point x under the model writes:

Original: [Equation]
Revised: [Equation]

Original:  As for VAEs, an evidence lower bound (ELBO) for this integral can be computed (Sohl-Dickstein et al., 2015; Kingma et al., 2021).
Revised: [Equation] As for VAEs, an evidence lower bound (ELBO) for this integral can be computed (Sohl-Dickstein et al., 2015; Kingma et al., 2021).

Original: It writes:
Revised: It writes:

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: (cid:123)(cid:122) (cid:125) Prior loss + T (cid:88) t = L t ( x ) (cid:124) (cid:123)(cid:122) (cid:125) Diffusion loss + E q

Original: [Equation] G | G [Equation]
Revised: ( G 1 | G ) [log p θ ( G | G 1 )]

Original: ] (cid:124) (cid:123)(cid:122) (cid:125) Reconstruction loss 
Revised: (cid:124) (cid:123)(cid:122) (cid:125) Reconstruction loss (14) with

Original: [Equation]
Revised: [Equation]

Original: 14) with

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: All these terms can be estimated: log p ( n G ) is computed using the frequencies of the number of nodes for each graph in the dataset.
Revised: All these terms can be estimated: log p ( n G ) is computed using the frequencies of the number of nodes for each graph in the dataset.

Original: The prior loss and the diffusion loss are KL divergences between categorical distribution, and the reconstruction loss is simply computed from the predicted probabilities for the clean graph given the last noisy graph G 1 .
Revised: The prior loss and the diffusion loss are KL divergences between categorical distribution, and the reconstruction loss is simply computed from the predicted probabilities for the clean graph given the last noisy graph G 1 .

Original: D P ROOFS
Revised: D P ROOFS

Original: Lemma 3.1: Equivariance
Revised: Lemma 3.1: Equivariance

Original: Proof.
Revised: Proof.

Original: Consider a graph G with n nodes, and π ∈ S n a permutation.
Revised: Consider a graph G with n nodes, and π ∈ S n a permutation.

Original: π acts trivially on y ( π. y
Revised: π acts trivially on y ( π. y

Original: =
Revised: =

Original: y ), it acts on X as π.
Revised: y ), it acts on X as π.

Original: X
Revised: X

Original: = π ′ X and on E as:
Revised: = π ′ X and on E as:

Original: [Equation]
Revised: [Equation]

Original: Let G t =
Revised: Let G t =

Original: ( X t , E t ) be a noised graph, and ( π. X t , π.
Revised: ( X t , E t ) be a noised graph, and ( π. X t , π.

Original: E t )
Revised: E t )

Original: its permutation.
Revised: its permutation.

Original: Then:
Revised: Then:

Original: • Our spectral and structural features are all permutation equivariant (for the node features) or invariant (for the graph level features): f ( π.
Revised: • Our spectral and structural features are all permutation equivariant (for the node features) or invariant (for the graph level features): f ( π.

Original: G t , t ) =
Revised: G t , t ) =

Original: π.f ( G t , t ) .
Revised: π.f ( G t , t ) .

Original: • The self-attention architecture is permutation equivariant.
Revised: • The self-attention architecture is permutation equivariant.

Original: The FiLM blocks are permutation equivariant, and the PNA pooling function is permutation invariant.
Revised: The FiLM blocks are permutation equivariant, and the PNA pooling function is permutation invariant.

Original: • Layer-normalization is permutation equivariant.
Revised: • Layer-normalization is permutation equivariant.

Original: DiGress is therefore the combination of permutation equivariant blocks.
Revised: DiGress is therefore the combination of permutation equivariant blocks.

Original: As a result, it is permutation equivariant: ϕ θ ( π.
Revised: As a result, it is permutation equivariant: ϕ θ ( π.

Original: G t , f ( π.
Revised: G t , f ( π.

Original: G t , t ))
Revised: G t , t ))

Original: =
Revised: =

Original: π.ϕ θ ( G t , f ( G t , t )) .
Revised: π.ϕ θ ( G t , f ( G t , t )) .

Original: Lemma 3.2: Invariant loss
Revised: Lemma 3.2: Invariant loss

Original: Proof.
Revised: Proof.

Original: It is important that the loss function be the same for each node and each edge in order to guarantee that
Revised: It is important that the loss function be the same for each node and each edge in order to guarantee that

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.3: Exchangeability
Revised: Lemma 3.3: Exchangeability

Original: Proof.
Revised: Proof.

Original: The proof relies on the result of Xu et al.
Revised: The proof relies on the result of Xu et al.

Original: (2022): if a distribution p ( G T ) is invariant to the action of a group G and the transition probabilities p ( G t − 1 | G t ) are equivariant, them p ( G 0 ) is invariant to the action of G .
Revised: (2022): if a distribution p ( G T ) is invariant to the action of a group G and the transition probabilities p ( G t − 1 | G t ) are equivariant, them p ( G 0 ) is invariant to the action of G .

Original: We apply this result to the special case of permutations:
Revised: We apply this result to the special case of permutations:

Original: • The limit noise distribution is the product of i i
Revised: • The limit noise distribution is the product of i.i.d.

Original: d. distributions on each node and edge.
Revised: distributions on each node and edge.

Original: It is therefore permutation invariant.
Revised: It is therefore permutation invariant.

Original: • The denoising neural networks is permutation equivariant.
Revised: • The denoising neural networks is permutation equivariant.

Original: • The function ˆ p θ ( G ) → p θ ( G t − 1 | G t )
Revised: • The function ˆ p θ ( G ) → p θ ( G t − 1 | G t )

Original: = (cid:80) G q
Revised: = (cid:80) G q

Original: ( G t − 1 , G | G t )ˆ p θ ( G ) defining the transition probabilities is equivariant to joint permutations of ˆ p θ ( G ) and G t .
Revised: ( G t − 1 , G | G t )ˆ p θ ( G ) defining the transition probabilities is equivariant to joint permutations of ˆ p θ ( G ) and G t .

Original: The conditions of (Xu et al., 2022) are therefore satisfied, and the model satisfies P ( X , E ) =
Revised: The conditions of (Xu et al., 2022) are therefore satisfied, and the model satisfies P ( X , E ) =

Original: P ( π. X , π. E ) for any permutation π .
Revised: P ( π. X , π. E ) for any permutation π .

Original: Theorem 4.1: Optimal prior distribution We first prove the following result:
Revised: Theorem 4.1: Optimal prior distribution We first prove the following result:

Original: Lemma D.1.
Revised: Lemma D.1.

Original: Let p be a discrete distribution over two variables.
Revised: Let p be a discrete distribution over two variables.

Original: It is represented by a matrix
Revised: It is represented by a matrix

Original: P ∈ R a × b .
Revised: P ∈ R a × b .

Original: Let m 1 and m 2 the marginal distribution of p : m 

Revised: Let m 1 and m 2 the marginal distribution of p : m

Original: i
Revised: i

Original: =
Revised: =

Original: (cid:80) bj =1 p ij and
Revised: (cid:80) bj =1 p ij and

Original: m 2 i =
Revised: m 2 i =

Original: (cid:80) ai =1 p ij .
Revised: (cid:80) ai =1 p ij .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We define L ( u , v ) :
Revised: We define L ( u , v ) :

Original: =
Revised: =

Original: || P − uv ′
Revised: || P − uv ′

Original: ||
Revised: ||

Original: 22 =
Revised: 22 =

Original: (cid:80) i,j ( p ij − u
Revised: (cid:80) i,j ( p ij − u

Original: i v j ) 2 .
Revised: i v j ) 2 .

Original: We derive this formula to obtain optimality conditions:
Revised: We derive this formula to obtain optimality conditions:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have ∂L ∂v j = 

Revised: Similarly, we have ∂L ∂v j =

Original: ⇐⇒ v j = (cid:80) i p ij u i / (cid:80)
Revised: ⇐⇒ v j = (cid:80) i p ij u i / (cid:80)

Original: j u 2 i .
Revised: j u 2 i .

Original: Since p , u and v are probability distributions, we have (cid:80) i,j p i,j = 1 , (cid:80) i u i
Revised: Since p , u and v are probability distributions, we have (cid:80) i,j p i,j = 1 , (cid:80) i u i

Original: = 1 and (cid:80)
Revised: = 1 and (cid:80)

Original: j v j = 1 .
Revised: j v j = 1 .

Original: Combining these equations, we have:
Revised: Combining these equations, we have:

Original: [Equation]
Revised: [Equation]

Original: So that:
Revised: So that:

Original: [Equation]
Revised: [Equation]

Original: and similarly v j 0 =
Revised: and similarly v j 0 =

Original: b i 0 .
Revised: b i 0 .

Original: Conversely, m 1 and m 2 belong to the set of feasible solutions.
Revised: Conversely, m 1 and m 2 belong to the set of feasible solutions.

Original: We have proved that the product distribution that is the closest to the true distribution of two variables is the product of marginals (for l 2 distance).
Revised: We have proved that the product distribution that is the closest to the true distribution of two variables is the product of marginals (for l 2 distance).

Original: We need to extend this result to a product (cid:81) ni
Revised: We need to extend this result to a product (cid:81) ni

Original: =1 u × (cid:81) 1 ≤ i,j ≤ n v of a distribution for nodes and a distribution for edges.
Revised: =1 u × (cid:81) 1 ≤ i,j ≤ n v of a distribution for nodes and a distribution for edges.

Original: We now view p as a tensor in dimension a n b n 2 .
Revised: We now view p as a tensor in dimension a n b n 2 .

Original: We denote p X the marginalisation of this tensor across the node dimensions ( p X ∈ R a n ), and p E the marginalisation across the edge dimensions ( p E ∈ R b n × n ) .
Revised: We denote p X the marginalisation of this tensor across the node dimensions ( p X ∈ R a n ), and p E the marginalisation across the edge dimensions ( p E ∈ R b n × n ) .

Original: By flattening the n first dimensions and the n 2 next, p can be viewed as a distribution over two variables (a distribution for the nodes and a distribution for the edges).
Revised: By flattening the n first dimensions and the n 2 next, p can be viewed as a distribution over two variables (a distribution for the nodes and a distribution for the edges).

Original: By application of our Lemma, p X and p E are the optimal approximation of p .
Revised: By application of our Lemma, p X and p E are the optimal approximation of p .

Original: However, p X is a joint distribution for all nodes and not the product (cid:81) ni
Revised: However, p X is a joint distribution for all nodes and not the product (cid:81) ni

Original: =1 u of a single distribution for all nodes.
Revised: =1 u of a single distribution for all nodes.

Original: We then notice that:
Revised: We then notice that:

Original: [Equation]
Revised: [Equation]

Original: for a function f that does not depend on u .
Revised: for a function f that does not depend on u .

Original: As (cid:80) i p
Revised: As (cid:80) i p

Original: Xi /n is exactly the empirical distribution of node types, the optimal u is the empirical distribution of node types as desired.
Revised: Xi /n is exactly the empirical distribution of node types, the optimal u is the empirical distribution of node types as desired.

Original: Overall, we have made two orthogonal projections: a projection from the distributions over graphs to the distributions over nodes, and a projection from the distribution over nodes to the product distributions u × · · · × u .
Revised: Overall, we have made two orthogonal projections: a projection from the distributions over graphs to the distributions over nodes, and a projection from the distribution over nodes to the product distributions u × · · · × u .

Original: Since the product distributions forms a linear space contained in the distributions over nodes, these two projections are equivalent to a single orthogonal projection from the distributions over graphs to the product distributions over nodes.
Revised: Since the product distributions forms a linear space contained in the distributions over nodes, these two projections are equivalent to a single orthogonal projection from the distributions over graphs to the product distributions over nodes.

Original: A similar reasoning holds for edges.
Revised: A similar reasoning holds for edges.

Original: D. S UBSTRUCTURE CONDITIONED GENERATION
Revised: D. S UBSTRUCTURE CONDITIONED GENERATION

Original: Given a subgraph S =
Revised: Given a subgraph S =

Original: ( X S , E S ) with n s nodes, we can condition the generation on S by masking the generated node and edge feature tensor at each reverse iteration step (Lugmayr et al., 2022).
Revised: ( X S , E S ) with n s nodes, we can condition the generation on S by masking the generated node and edge feature tensor at each reverse iteration step (Lugmayr et al., 2022).

Original: As our model is permutation equivariant, it does not matter what entries are masked: we therefore choose the first n s ones.
Revised: As our model is permutation equivariant, it does not matter what entries are masked: we therefore choose the first n s ones.

Original: After sampling G t − 1 , we update X and E using
Revised: After sampling G t − 1 , we update X and E using

Original: [Equation]
Revised: [Equation]

Original: where M X ∈ R n × a and M E ∈ R n × n × b are masks indicating the n s first nodes.
Revised: where M X ∈ R n × a and M E ∈ R n × n × b are masks indicating the n s first nodes.

Original: In Figure 8, we showcase an example for molecule generation: we follow the setting proposed by (Maziarz et al., 2022) and generate molecules starting from a particular motif called 1,4-Dihydroquinoline 1 .
Revised: In Figure 8, we showcase an example for molecule generation: we follow the setting proposed by (Maziarz et al., 2022) and generate molecules starting from a particular motif called 1,4-Dihydroquinoline 1 .

Original: [Table]
Revised: [Table]

Original: E E XPERIMENTAL DETAILS AND ADDITIONAL RESULTS E. A BSTRACT GRAPH GENERATION
Revised: E E XPERIMENTAL DETAILS AND ADDITIONAL RESULTS E. A BSTRACT GRAPH GENERATION

Original: Metrics The reported metrics compare the discrepancy between the distribution of some metrics on a test set and the distribution of the same metrics on a generated graph.
Revised: Metrics The reported metrics compare the discrepancy between the distribution of some metrics on a test set and the distribution of the same metrics on a generated graph.

Original: The metrics measured are degree distributions, clustering coefficients
Revised: The metrics measured are degree distributions, clustering coefficients

Original: [Table]
Revised: [Table]

Original: and orbit counts (it measures the distribution of all substructures of size 4).
Revised: and orbit counts (it measures the distribution of all substructures of size 4).

Original: We do not report raw numbers but ratios computed as follows:
Revised: We do not report raw numbers but ratios computed as follows:

Original: [Equation]
Revised: [Equation]

Original: Community-20 In Table 5, we also provide results for the smaller Community-20 dataset which contains 200 graphs drawn from a stochastic block model with two communities.
Revised: Community-20 In Table 5, we also provide results for the smaller Community-20 dataset which contains 200 graphs drawn from a stochastic block model with two communities.

Original: We observe that
Revised: We observe that

Original: DiGress performs very well on this small dataset.
Revised: DiGress performs very well on this small dataset.

Original: E. QM 
Revised: E. QM

Original: Metrics Because it is the metric reported in most papers, the validity metric we report is computed by building a molecule with RdKit and trying to obtain a valid SMILES string out of it.
Revised: Metrics Because it is the metric reported in most papers, the validity metric we report is computed by building a molecule with RdKit and trying to obtain a valid SMILES string out of it.

Original: As explained by Jo et al.
Revised: As explained by Jo et al.

Original: (2022), this method is not perfect because QM9 contains some charged molecules which would be considered as invalid by this method.
Revised: (2022), this method is not perfect because QM9 contains some charged molecules which would be considered as invalid by this method.

Original: They thus compute validity using a more relaxed definition that allows for some partial charges, which gives them a small advantage.
Revised: They thus compute validity using a more relaxed definition that allows for some partial charges, which gives them a small advantage.

Original: Ablation study We perform an ablation study in order to highlight the role of marginal transitions and auxiliary features.
Revised: Ablation study We perform an ablation study in order to highlight the role of marginal transitions and auxiliary features.

Original: In this setting, we also measure atom stability and molecule stability as defined in (Hoogeboom et al., 2022).
Revised: In this setting, we also measure atom stability and molecule stability as defined in (Hoogeboom et al., 2022).

Original: Results are presented in Figure 6.
Revised: Results are presented in Figure 6.

Original: Novelty We follow Vignac & Frossard (2021) and don’t report novelty for QM9 in the main table.
Revised: Novelty We follow Vignac & Frossard (2021) and don’t report novelty for QM9 in the main table.

Original: The reason is that since QM9 is an exhaustive enumeration of the small molecules that satisfy a given set of constrains, generating molecules outside this set is not necessarily a good sign that the network has correctly captured the data distribution.
Revised: The reason is that since QM9 is an exhaustive enumeration of the small molecules that satisfy a given set of constrains, generating molecules outside this set is not necessarily a good sign that the network has correctly captured the data distribution.

Original: For the interested reader, DiGress achieves on average a novelty of 33 . 4% on QM9 with implicit hydrogens, while ConGress obtains 40 .
Revised: For the interested reader, DiGress achieves on average a novelty of 33 . 4% on QM9 with implicit hydrogens, while ConGress obtains 40 .

Original: E. MOSES AND G UACA M OL
Revised: E. MOSES AND G UACA M OL

Original: Datasets For both MOSES and GuacaMol, we convert the generated graphs to SMILES using the code of Jo et al.
Revised: Datasets For both MOSES and GuacaMol, we convert the generated graphs to SMILES using the code of Jo et al.

Original: (2022) that allows for some partial charges.
Revised: (2022) that allows for some partial charges.

Original: We note that GuacaMol contains complex molecules that are difficult to process, for example because they contain formal charges or fused rings.
Revised: We note that GuacaMol contains complex molecules that are difficult to process, for example because they contain formal charges or fused rings.

Original: As a result, mapping the train smiles to a graph
Revised: As a result, mapping the train smiles to a graph

Original: [Table]
Revised: [Table]

Original: and then back to a train SMILES does not work for around 20% of the molecules.
Revised: and then back to a train SMILES does not work for around 20% of the molecules.

Original: Even if our model is able to correctly model these graphs and generate graphs that are similar, these graphs cannot be mapped to SMILES strings to be evaluated by GuacaMol.
Revised: Even if our model is able to correctly model these graphs and generate graphs that are similar, these graphs cannot be mapped to SMILES strings to be evaluated by GuacaMol.

Original: More efficient tools for processing complex molecules as graphs are therefore needed to truly achieve good performance on this dataset.
Revised: More efficient tools for processing complex molecules as graphs are therefore needed to truly achieve good performance on this dataset.

Original: Metrics Since MOSES and Guacamol are benchmarking tools, they come with their own set of metrics that we use to report the results.
Revised: Metrics Since MOSES and Guacamol are benchmarking tools, they come with their own set of metrics that we use to report the results.

Original: We briefly describe this metrics: Validity measures the proportion of molecules that pass basic valency checks.
Revised: We briefly describe this metrics: Validity measures the proportion of molecules that pass basic valency checks.

Original: Uniqueness measures the proportion of molecules that have different SMILES strings (which implies that they are non-isomorphic).
Revised: Uniqueness measures the proportion of molecules that have different SMILES strings (which implies that they are non-isomorphic).

Original: Novelty measures the proportion of generated molecules that are not in the training set.
Revised: Novelty measures the proportion of generated molecules that are not in the training set.

Original: The filter score measures the proportion of molecules that pass the same filters that were used to build the test set.
Revised: The filter score measures the proportion of molecules that pass the same filters that were used to build the test set.

Original: The Frechet ChemNetDistance (FCD) measures the similarity between molecules in the training set and in the test set using the embeddings learned by a neural network.
Revised: The Frechet ChemNetDistance (FCD) measures the similarity between molecules in the training set and in the test set using the embeddings learned by a neural network.

Original: SNN is the similarity to a nearest neighbor, as measured by Tanimoto distance.
Revised: SNN is the similarity to a nearest neighbor, as measured by Tanimoto distance.

Original: Scaffold similarity compares the frequencies of
Revised: Scaffold similarity compares the frequencies of

Original: Bemis-Murcko scaffolds.
Revised: Bemis-Murcko scaffolds.

Original: The KL divergence compares the distribution of various physicochemical descriptors.
Revised: The KL divergence compares the distribution of various physicochemical descriptors.

Original: Likelihood Since other methods did not report likelihood for GuacaMol and MOSES, we did not include our NLL results in the table neither.
Revised: Likelihood Since other methods did not report likelihood for GuacaMol and MOSES, we did not include our NLL results in the table neither.

Original: We obtain a test NLL of 23.44 on MOSES (on the separate scaffold test set) and 23.65 on GuacaMol.
Revised: We obtain a test NLL of 23.44 on MOSES (on the separate scaffold test set) and 23.65 on GuacaMol.

Original: F S AMPLES FROM OUR MODEL
Revised: Size extrapolation While the vast majority of molecules in QM9 have the same number of atoms, molecules in MOSES and Guacamol have varying sizes. On these datasets, we would like to know if DiGress can generate larger molecules than it has been trained on. This problem is usually called size extrapolation in the graph neural network literature. To measure the network ability to extrapolate, we set the number of atoms to generate to n max + k , where n max is the maximal graph size in the dataset and k ∈ [5 , 10 , 20] .

Revised: We generate 24 batches of 256 molecules (=6144 molecules) in each setting and measure the proportion of valid and unique molecules – all these molecules are novel since they are larger than the training set.

Revised: The results are presented in Table 7.

Revised: We observe an important discrepancy between the two datasets:

Revised: DiGress is very capable of extrapolation on GuacaMol, but completely fails on MOSES.

Revised: This can be explained by the respective statistics of the datasets: MOSES features molecules that are relatively homogeneous in size.

Revised: On the contrary, GuacaMol features molecules that are much larger than the dataset average.

Revised: The network is therefore trained on more diverse examples, which we conjecture is why it learns some size invariance properties.

Revised: The major difference in extrapolation ability that we obtain clearly highlights the value of large and diverse datasets.

Revised: We finally note that our denoising network was not designed to be size invariant, as it for example features sum aggregation functions at each layer.

Revised: Specific techniques such as SizeShiftReg (Buffelli et al., 2022) could also be used to improve the size-extrapolation ability of DiGress if needed for downstream applications.

Revised: F S AMPLES FROM OUR MODEL

Revised: [Table]