Paper: A Simple and Scalable Representation for Graph Generation (ICLR-2024)

Reviewer comment (Presentation):
There is a problem in Appendix C.1 Table 11 (b) Large graphs (|V|max ≤ 187).

Author response:
**W1. There is a problem in Appendix C.1 Table 11 (b) Large graphs (|V|max ≤ 187).**

Thank you for pointing out the typo. We fixed this to $399 \leq |V|_{\max} \leq 5037$ in our manuscript.

Reviewer comment (Evaluation):
For the molecule generation task, it is important to generate some novel molecules for further filtering in drag design and other real-world scenarios. This paper does not evaluate the portion of the novel-generated molecules in Table 4. In the extreme case, the model may only be able to generate molecules in the training set.

Author response:
**W2. Provide the novelty and uniqueness rates achieved by GEEL in the molecular generation experiments.**

Thank you for pointing out additional metrics for molecular graph generation. We provide the novelty and uniqueness in the following table and updated the performance of molecular graph generation in our manuscript. One can observe that our GEEL shows competitive novelty and uniqueness compared to the baselines. We updated Table 16 in Appendix F of our manuscript.


|                 |             |   QM9 |       |  ZINC |        |
| --------------- | ----------- | -----:| -----:| -----:| ------:|
|                 |             | Uniq. |  Nov. | Uniq. |   Nov. |
| Molecule        | MoFlow      | 98.65 | 94.72 | 99.99 | 100.00 |
|                 | STGG        | 96.76 | 72.73 | 99.99 |  99.89 |
| Domain-agnostic | DiGress     | 96.67 | 25.58 | 99.97 |  99.89 |
|                 | DruM        | 96.90 | 24.15 | 99.97 |  99.99 |
|                 | GEEL (Ours) | 96.08 | 22.30 | 99.97 |  99.98 |

We note that the models make a tradeoff between the quality (e.g., NSPDK and FCD) and novelty of the generated graph since the graph generative models that faithfully learn the distribution put a high likelihood on the training dataset. In particular, the tradeoff is more significant in QM9 due to the large dataset size (134k) compared to the relatively small search space (molecular graphs with only up to nine heavy atoms). We also note that one can compensate for the non-unique and non-novel molecules by filtering them out and generating new molecules.

Reviewer comment (Experiments):
It is hard to claim that GEEL outperforms BiGG. In Table 1, the performance is close. In terms of speed, BiGG runs on GeForce GTX 1080 Ti while GEEL runs on GeForce RTX 3090, which makes Table 2’s result unfair.

Author response:
**W3. It is hard to claim that GEEL outperforms BiGG. In Table 1, the performance is close. In terms of speed, BiGG runs on GeForce GTX 1080 Ti while GEEL runs on GeForce RTX 3090, which makes Table 2’s result unfair.**

Thank you for pointing out the unclarity. We conducted the inference time analysis in Table 2 on the same computational environment (GTX 1080 Ti) for all the models including GEEL and BiGG. We have clarified this in our updated manuscript in Appendix B.3.

Hence, one can safely conclude that GEEL is faster than BiGG. Furthermore, we claim that GEEL slightly outperforms BiGG in Table 1 since GEEL is marked bold for 20 out of 24 metrics in Table 1, while BiGG is marked bold for 14 out of 24 metrics. As mentioned in Section 4.1, we mark the numbers in bold if it is comparable to the metrics of the training graphs.

Reviewer comment (Experiments):
If this paper includes some ablation study of replacing the intra- and inter-edge gap encoding with traditional encodings, it will be better.

Author response:
**Q1. If this paper includes some ablation study of replacing the intra- and inter-edge gap encoding with traditional encodings, it will be better. Table 6 could include more experiment results. The parameter size can be adjusted to avoid Out-Of-Memory.**

The ablation study of replacing the intra- and inter-edge gap with traditional encodings (i.e., original edge list) is in Table 6.  The edge list (second row) indicates the traditional encodings and the edge list + intra gap (third row) indicates employing only intra-edge gap. We updated our manuscript to clarify what each row in Table 6 means.

Notably, the traditional encoding encounters out-of-memory in our computational environment even after adjusting the parameters or even setting the batch size to 1. This supports the superiority of our GEEL, concerning the memory requirement.

Reviewer comment (Experiments):
Table 6 could include more experiment results.

Author response:
**Q1. If this paper includes some ablation study of replacing the intra- and inter-edge gap encoding with traditional encodings, it will be better. Table 6 could include more experiment results. The parameter size can be adjusted to avoid Out-Of-Memory.**

The ablation study of replacing the intra- and inter-edge gap with traditional encodings (i.e., original edge list) is in Table 6.  The edge list (second row) indicates the traditional encodings and the edge list + intra gap (third row) indicates employing only intra-edge gap. We updated our manuscript to clarify what each row in Table 6 means.

Notably, the traditional encoding encounters out-of-memory in our computational environment even after adjusting the parameters or even setting the batch size to 1. This supports the superiority of our GEEL, concerning the memory requirement.

Reviewer comment (Experiments):
The parameter size can be adjusted to avoid Out-Of-Memory.

Author response:
**Q1. If this paper includes some ablation study of replacing the intra- and inter-edge gap encoding with traditional encodings, it will be better. Table 6 could include more experiment results. The parameter size can be adjusted to avoid Out-Of-Memory.**

The ablation study of replacing the intra- and inter-edge gap with traditional encodings (i.e., original edge list) is in Table 6.  The edge list (second row) indicates the traditional encodings and the edge list + intra gap (third row) indicates employing only intra-edge gap. We updated our manuscript to clarify what each row in Table 6 means.

Notably, the traditional encoding encounters out-of-memory in our computational environment even after adjusting the parameters or even setting the batch size to 1. This supports the superiority of our GEEL, concerning the memory requirement.

Reviewer comment (Evaluation):
Results on Uniqueness and Novelty seem to be missing. It is not clear whether the generated graphs have duplicacy. The reviewer refers to the metrics section of [A] for details, and suggests that adding these metrics (at least for a few datasets) could further improve the quality of the manuscript.

Author response:
**W2/Q2. Results on uniqueness and novelty metrics seem to be missing.**

Thank you for such an insightful comment! Following your suggestion, we evaluated the validity, novelty, and uniqueness of GEEL and the considered baselines. We report the new results in Table 15 and Table 16 of Appendix F. We think our empirical results are now much more comprehensive and reliable, thanks to your review.

In our new experiments, GEEL achieves the 30\% \~ 90\% novelty, which is better than autoregressive models like GRAN and BiGG, but lower than the diffusion-based graph generative models as the reviewer anticipated. This is partially due to the inherent trade-off between novelty and the capability of generative models to faithfully learn the data distribution. Even diffusion models suffer from this trade-off, e.g., DiGress achieves 30\% novelty for the QM9 dataset. We also note that this trade-off is less severe for harder problems, e.g., point cloud and ZINC250k, which is the main target of our work.

Finally, we note the minor modification of GEEL in the new experiments. Our GEEL now samples a new C-M ordering for a graph at each training step (instead of fixing a unique ordering per graph). We found this to improve novelty without a decrease in performance and changing the architecture. We updated all of our experimental results to incorporate this change. Note that the results for the Ego dataset are to be updated, as the experiments have not yet been completed.

Reviewer comment (Experiments):
The scalability analysis does not seem to be complete. The reviewer asks the authors to clarify how scalability results are computed, for example, whether the batch size is set to 1. Since GraphGen [A] also works at the edge list level O(#Edges), the reviewer expects the authors to compare with GraphGen for scalability comparison. The reviewer also asks the authors to justify why generation time is shown for one graph, raising the concern that it could be an outlier, as only a single number is provided without any standard deviation. They suggest that it might make more sense to generate a batch of graphs and report mean and standard deviation.

Author response:
**W3/Q3-1. Could the authors clarify how scalability results are computed?**

We conducted inference time analysis by setting the batch size to 10 and dividing the total inference time by 10 to compute the generation time for a single graph. Therefore, the reported inference time is not an outlier that shows the time for a single graph generation. We updated our manuscript to clarify this in Appendix B.

**Q3-2. Since GraphGen works at the edge list level, I would expect the authors to compare with GraphGen in Table 6.**

To incorporate your suggestion, we will add a comparison with GraphGen in our future manuscript. Please understand that we are unable to provide the result during the rebuttal phase. Unfortunately, we currently do not have the computational environment (GeForce GTX 1080 Ti) used for Table 6, which is necessary for a fair comparison.

Reviewer comment (Novelty):
perhaps this article is not well-suited for ICLR, since the focus should be on learning representations, but this paper does not revolve around representation learning. Again, I don't think this is fatal, and I would like to hear how this work places itself in the context of this conference by the authors.

Author response:
**W1. The work is perhaps not well-suited for ICLR since it does not include any learning representations.**

Thank you for checking this. We would like to point out that ICLR 2024 mentions "generative models" as one of the relevant topics in its call for papers (https://iclr.cc/Conferences/2024/CallForPapers). Hence we believe our work (graph generative model) aligns with the context of ICLR.

We also provide a few examples of similar works in previous ICLRs:
- Shi, C., et al., GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation, ICLR 2020.
- Liu, M., et al., GraphEBM: Molecular Graph Generation with Energy-Based Models, ICLR 2021.
- Vignac, C., et al., DiGress: Discrete Denoising diffusion for graph generation, ICLR 2023.

Reviewer comment (Presentation):
What do the asterisks placed after Graphgen and Graphgen-redux in Table 1 mean?

Author response:
**Q1. What do the asterisks placed after Graphgen and Graphgen-redux in Table 1 mean?**

Thank you for pointing out this typo. The asterisks indicated edge list-based graph generative models, but we deleted them since they are not very informative. We instead mention that Graphgen and Graphgen-redux are edge list-based graph generation methods in Section 4.

Reviewer comment (Writing):
What is meant by "comparable" MMD? Which criteria is used to define two MMDs comparable?

Author response:
**Q2. What is meant by "comparable" MMD? Which criteria is used to define two MMDs comparable?**

The comparability of MMD values is determined by examining whether the MMD falls within a range of one standard deviation (reported in Table 11 of Appendix C.1). This was explained in the evaluation protocol in Section 4.1, but we also added this to the caption of Table 1 to further be more clear.

Reviewer comment (Reproducibility):
In Figure 4, what is the value of the $c_1$ and $c_2$ constants? Have they been explicitly calculated?

Author response:
**Q3. In Figure 4, what is the value of the c1 and c2 constants? Have they been explicitly calculated?**

The constants $c_1$ and $c_2$ in Figure 4 are 0.5 and 2, respectively. We added the lines for visual alignment between the inference time and the theoretical computational complexity $O(M)$, where $M$ is the number of edges. We tried a few values to select the visually plausible constants.

Reviewer comment (Experiments):
Speaking of which, I think it would be better to add the performances of the method in the MOSES benchmark. It should be fairly doable in a short time since the method is fast both in training and inference.

Author response:
**Q5. Add additional MOSES dataset for molecular graph generation.**

Thank you for pointing out the meaningful benchmark. We provide the molecular generation performance on the MOSES dataset in the following table and will incorporate this into our future manuscript. We can observe that GEEL shows competitive performance to molecule-specific methods and superior performance to the domain-agnostic method, DiGress. We reported the new results in Table 14 of Appendix E.


|              |   Method    |     Val. ↑ | NSPDK ↓ |      FCD ↓ |    Scaf. ↑ |     SNN  ↑ |    Frag. ↑ | Uniq.  ↑ | Nov. ↑ |
|:------------:|:-----------:| ----------:| -------:| ----------:| ----------:| ----------:| ----------:| -------:| -------:|
|   **Molecule-specific**   |   CharRNN   |      97.48 |       - |     0.0732 |     0.9242 |     0.6015 |     0.9998 |    0.9909     |     0.8419    |
|              |   JT-VAE    | 100.00 |       - |     0.3954 |     0.8964 |     0.5477 |     0.9965 |    0.9996     |     0.9143    |
|              |    STGG     | 100.00 |       - | 0.0680 | 0.9416 | 0.6359 | 0.9998 |   0.9987      |     0.6727    |
| **Domain-agnostic** |   DiGress   | 100.00 |  0.0005 |     0.6788 |     0.8653 |     0.5488 |     0.9988 |    0.9999     |    0.9447     |
|              | GEEL (Ours) |    100.00        |    0.0001     |      0.1528      |      0.8692      |       0.6273     |     0.9996       |    0.9989     |  0.7313       |

Reviewer comment (Reproducibility):
What are the vocabulary sizes for the molecular generation experiments (on QM9 and ZINC250k)?

Author response:
**Q6. What are the vocabulary sizes for the molecular generation experiments (on QM9 and ZINC250k)?**

The vocabulary size of the molecules is $2B+|X|+|E|$, where $B$ denotes the bandwidth, $|X|$ denotes the number of node features, and $|E|$ denotes the number of edge features. The bandwidths, the number of nodes, and the number of edges of QM9 and ZINC250k are provided in the following table, which are also provided in Appendix A and C in our original manuscript. The vocabulary sizes of QM9 and ZINC250k are 27 and 53, respectively. We updated our manuscript in Section 3.3.


|          | $B$ | $X$ | $E$ | Vocab. |
|:--------:|:---:|:---:|:---:|:------:|
|   **QM9**    |  5  | 13  |  4  |   27   |
| **ZINC250k** | 10  | 29  |  4  |   53   |