Paper: Directed Graph Generation with Heat Kernels (ICLR-2024)

Reviewer comment (Writing):
The decision to set M to be a constant matrix should be further motivated and explained (to people familiar with this, this is a natural choice, but this can be unclear to the less initiated readers).

Author response:
Thank you for the suggestion. We have added in Section 2 the motivation of our formulation of $\textbf{M}$, which is to represent a non-informative matrix when the node representation matrix is constrained to be column stochastic.  Each column of $\textbf{M}$ is defined as the uniform probability distribution vector, which corresponds to the maximum entropy probability distribution vector.  
Each column of $\textbf{M}$ also corresponds to the expected value of a random variable following a flat Dirichlet distribution.

Reviewer comment (Writing):
Crucial concepts rely on very recent work such as the Set Transformer in 2019 and the work of Veerman and Lyons in 2020. This makes the article more difficult to read. I suggest that the authors try their best to make this work more self contained in the presentation.

Author response:
Thank you for this suggestion. We have added a high-level description of SetTransformers and our motivation to use it (i.e. using a model that is invariant to the order of the rows of our node representations) in Section 3.1. The necessary concepts from Veerman and Lyons are given in our submission. We cite their paper only when (1) we define the (negative of) the random walk Laplacian matrix, which is a standard definition in graph theory, and when (2) we say that its matrix exponential is column stochastic, which can be verified by eigendecomposition. This last concept is actually explained in our Appendix F when we study the eigendecomposition of our heat kernel matrix. We have then added a reference to Appendix F for details.

Reviewer comment (Experiments):
The empirical evaluation is limited to very simple models (ER and SBM) under the squared MMD distance for various descriptors, under hyperparameter settings of 3 blocks and p as 0.6, seemingly without much justification.

Author response:
We would like to emphasize that the blocks in the stochastic block models are disconnected only in the qualitative experiment of Section 5.3, so that the different modes are easy to visualize. In our experiments with quantitative results (i.e. Section 5.2 and Appendix H.2), the different blocks are still connected but with low transition probability between blocks. The probabilities of transition between blocks are given in Equation (8) and Equation (30).  
Therefore, our evaluation metrics capture the fine-grained properties of connected digraphs.  
Moreover, we provide additional experiments with larger graphs in Appendix H.2 with $p = 0.4$ for the Erdos-Renyi category, and 5 blocks for the stochastic block models.

Reviewer comment (Evaluation):
Is there a concrete reason/justification for why the empirical evaluation is so focused on disconnected digraphs? My impression is that many interesting applications concern connected/strongly connected digraphs. Is there a possibility where the evaluation metrics (clustering coefficients etc) are just capturing the disjointness of the generated graph, rather than the more fine-grained properties of connectivity within a single connected/strongly connected digraph?

Author response:
We would like to emphasize that the blocks in the stochastic block models are disconnected only in the qualitative experiment of Section 5.3, so that the different modes are easy to visualize. In our experiments with quantitative results (i.e. Section 5.2 and Appendix H.2), the different blocks are still connected but with low transition probability between blocks. The probabilities of transition between blocks are given in Equation (8) and Equation (30).  
Therefore, our evaluation metrics capture the fine-grained properties of connected digraphs.  
Moreover, we provide additional experiments with larger graphs in Appendix H.2 with $p = 0.4$ for the Erdos-Renyi category, and 5 blocks for the stochastic block models.

Reviewer comment (Presentation):
The overall writing and organization could be improved. In particular, section 3, which lack continuity.

Author response:
Thank you for the suggestion. We have shortened the related work section by moving the similarities with diffusion models to the Appendix, and we have rewritten some parts of Section 3 that might be confusing. 
We have also included different variance parameters ($\sigma^2 = 100, 10, 1$) in Table 2 as requested. As a random method, we have added results when sampling each column of the input of the decoder directly from a flat Dirichlet distribution. We have tested the sampling method that replaces line 2 of algorithm 1 by sampling the non-diagonal elements of the adjacency matrix uniformly in the interval [0,1] (while keeping the diagonal elements equal to 1). We call that approach "continuous DGDK" because the sampling space is continuous as opposed to our previous sampling approach that is discrete. The former random method outperforms the baseline but is outperformed by our approach. On the other hand, the latter is competitive with our proposed sampling approach since it also exploits the learned matrix $\textbf{N}$. Due to lack of time, we did not include standard deviation, we intend to do it in the next version.

Reviewer comment (Experiments):
The experiments are a bit limited; the authors should report the standard deviation in the table.

Author response:
Thank you for the suggestion. We have shortened the related work section by moving the similarities with diffusion models to the Appendix, and we have rewritten some parts of Section 3 that might be confusing. 
We have also included different variance parameters ($\sigma^2 = 100, 10, 1$) in Table 2 as requested. As a random method, we have added results when sampling each column of the input of the decoder directly from a flat Dirichlet distribution. We have tested the sampling method that replaces line 2 of algorithm 1 by sampling the non-diagonal elements of the adjacency matrix uniformly in the interval [0,1] (while keeping the diagonal elements equal to 1). We call that approach "continuous DGDK" because the sampling space is continuous as opposed to our previous sampling approach that is discrete. The former random method outperforms the baseline but is outperformed by our approach. On the other hand, the latter is competitive with our proposed sampling approach since it also exploits the learned matrix $\textbf{N}$. Due to lack of time, we did not include standard deviation, we intend to do it in the next version.

Reviewer comment (Experiments):
The experiments are a bit limited; it is important to report other distances than the current MMD with RBF kernel, either with different kernels and/or with different variance parameters.

Author response:
Thank you for the suggestion. We have shortened the related work section by moving the similarities with diffusion models to the Appendix, and we have rewritten some parts of Section 3 that might be confusing. 
We have also included different variance parameters ($\sigma^2 = 100, 10, 1$) in Table 2 as requested. As a random method, we have added results when sampling each column of the input of the decoder directly from a flat Dirichlet distribution. We have tested the sampling method that replaces line 2 of algorithm 1 by sampling the non-diagonal elements of the adjacency matrix uniformly in the interval [0,1] (while keeping the diagonal elements equal to 1). We call that approach "continuous DGDK" because the sampling space is continuous as opposed to our previous sampling approach that is discrete. The former random method outperforms the baseline but is outperformed by our approach. On the other hand, the latter is competitive with our proposed sampling approach since it also exploits the learned matrix $\textbf{N}$. Due to lack of time, we did not include standard deviation, we intend to do it in the next version.

Reviewer comment (Experiments):
The experiments are a bit limited; in the tables, it is hard to understand the magnitude of the scores. It would be great to add a row for a random method (e.g. random adjacency matrix used in line 2 of Alg.1).

Author response:
Thank you for the suggestion. We have shortened the related work section by moving the similarities with diffusion models to the Appendix, and we have rewritten some parts of Section 3 that might be confusing. 
We have also included different variance parameters ($\sigma^2 = 100, 10, 1$) in Table 2 as requested. As a random method, we have added results when sampling each column of the input of the decoder directly from a flat Dirichlet distribution. We have tested the sampling method that replaces line 2 of algorithm 1 by sampling the non-diagonal elements of the adjacency matrix uniformly in the interval [0,1] (while keeping the diagonal elements equal to 1). We call that approach "continuous DGDK" because the sampling space is continuous as opposed to our previous sampling approach that is discrete. The former random method outperforms the baseline but is outperformed by our approach. On the other hand, the latter is competitive with our proposed sampling approach since it also exploits the learned matrix $\textbf{N}$. Due to lack of time, we did not include standard deviation, we intend to do it in the next version.

Reviewer comment (Experiments):
The experiments are a bit limited; the results in 5.3 could be in the main body of the text by shortening other sections (e.g. related work).

Author response:
Thank you for the suggestion. We have shortened the related work section by moving the similarities with diffusion models to the Appendix, and we have rewritten some parts of Section 3 that might be confusing. 
We have also included different variance parameters ($\sigma^2 = 100, 10, 1$) in Table 2 as requested. As a random method, we have added results when sampling each column of the input of the decoder directly from a flat Dirichlet distribution. We have tested the sampling method that replaces line 2 of algorithm 1 by sampling the non-diagonal elements of the adjacency matrix uniformly in the interval [0,1] (while keeping the diagonal elements equal to 1). We call that approach "continuous DGDK" because the sampling space is continuous as opposed to our previous sampling approach that is discrete. The former random method outperforms the baseline but is outperformed by our approach. On the other hand, the latter is competitive with our proposed sampling approach since it also exploits the learned matrix $\textbf{N}$. Due to lack of time, we did not include standard deviation, we intend to do it in the next version.

Reviewer comment (Writing):
In eq.1, you also need to specify the initial condition at \(t=0\). You could also explain \(X(0)\) has \(d\) signals that you diffuse following the heat equation (it might give more intuition).

Author response:
Thank you for the suggestion. We have included that intuition after Eq. (2). All the necessary conditions/constraints are included in Eq. (1) so that Eq. (2) is its solution (see e.g. Veermans and Lyons).

Reviewer comment (Writing):
Regarding the sentence “Finally, some denoising decoder is trained to predict the nodes and/or edges when given only \(X(T)\) as input (see details in Section 3.1)”: I don't fully understand this sentence. Looking at \(\mathcal{L}_{node}\) it is trained to reverse the diffusion process. It is not trained to predict the node, but rather the initial \(d\) signals.

Author response:
Your understanding is correct. Our decoders are trained to reconstruct the initial representation that does not contain noise. We have replaced the word "predict" with "reconstruct".

Reviewer comment (Reproducibility):
Is the decoder conditioned on the noise level (e.g. like in score matching) ?

Author response:
The decoder is in a sense conditioned on the "noise level" since we formulate: (1) our encoder so that its limit tends to $\textbf{M}$ when $T$ tends to infinity, (2) each column of $\textbf{M}$ corresponds to the mean/expected value of a random variable following the flat Dirichlet distribution (with all the parameters $\alpha$ equal to 1). 
Therefore, the input of our decoder is "close" to the expected value of the flat Dirichlet distribution. However, this is different from the noise level conditioning in regular diffusion models trained with denoising score matching at various noise levels. In those cases, the decoder, or denoiser, is trained at multiple different noise levels simultaneously. But in our case, our decoder is only trained at one, "the final", noise level. Since our method is an efficient one-shot generation technique, the decoder never needs to be trained or evaluated at "intermediate" noise levels, as in diffusion models. Hence, an explicit noise level conditioning like in diffusion models is not necessary for our method.

Reviewer comment (Experiments):
The experimental evaluation is a bit substandard given the relatively large recent literature on the topic. The authors should at least match SPECTRE (cited) for the evaluation protocol.

Author response:
We agree that there is a large literature about graph generation. However, the literature focuses mostly on undirected graphs, whereas our paper focuses on directed graphs. All our training graphs are directed, including Erdos-Renyi or stochastic block models. We explain in Section 4 how Spectre cannot be applied to directed graphs as their method assumes that their Laplacian matrix lies on a Stiefel manifold, which is not the case when the graph is directed (i.e. the adjacency matrix is not symmetric).

Reviewer comment (Theory):
While it is clear that the link predictor tries to match what it has seen in the training set, it is not clear how their approach changes the Erdos-Renyi distribution to one more similar to the training set.

Author response:
About the evaluation on the Erdos-Renyi distribution category specifically, we use a probability of $p = 0.6$ in Table 1, and $p=0.4$ in Table 2 to generate the training digraphs in this category. In those tables, we report the evaluation metric based on the in-degree MMD between the training and generated sets. 
Even in the directed case, the Erdos-Renyi category is defined by the overall in-degree of nodes. Our approach results in low MMD for that metric, which means that in-degrees between training and generated graphs are similar.