Paper: MAGNet: Motif-Agnostic Generation of Molecules from Shapes (ICLR-2024)

Reviewer comment (Novelty):
Not including a related work section discussing the connection between MAGNet and existing shape-based molecular generation methods [1, 2, 3].

Author response:
> Connection between our work and other "shape" works that were not discussed

- Note that the mentioned works focus on a different task where shapes refer to surfaces in 3D space while we consider untyped 2D graph as shapes. We are familiar with these works but originally refrained from including them, to avoid confusion with respect to the task we address. Adams et al., in particular, use MoLeR to create sufficient priors for their method. The same is possible with our approach and, due to the introduction of shapes, even with more flexibility.
We realise that not including these works also has the potential for confusion. Hence, we differentiate our methods from the mentioned "shape" works in the updated version of the related work section, cf. end of Section 3.

Reviewer comment (Writing):
The proposed method is not very clear to me. See details in Questions.

Author response:
> I still find the paper challenging to grasp due to the scattered presentation of details.

Based on the feedback we received, we believe that our (revised) manuscript follows a clear structure. Hence, we are unsure about the scattered details that were mentioned in the feedback. If the criticism is related to referring to the appendix multiple times from the main manuscript, we would like to clarify that the appendix A only provides additional examples and supporting information that are not crucial for the main part of the manuscript and were asked for in the initial review. The same applies, for example, to the appendix sections B which provides implementation details that will become less relevant once the code is publicly available. We hope this clears up any confusion. 

> An improved overview section could be beneficial in this regard.

We would like to understand why the provided overview paragraphs are insufficient. We include multiple overview paragraphs throughout the main manuscript to help guide the reader: 
- In the introduction, we clearly state our contributions, followed by a detailed explanation of the claims made in the paper and how they are reflected in the experiment section. 
- Section 2 introduces our contributions on the methodological level by first discussing the data factorisation and nomenclature, followed by our fragmentation procedure and MAGNet as a generative model fitting the fragmentation and proposed factorisation. This structure is again highlighted at the beginning of Section 2. 
- In Section 3, after discussing key aspects of our manuscript in two paragraphs of related work, we put MAGNet in a broader context and distinguish our work from others which also was refined based on the feedback we got in the the initial review. 
- Section 4, the experiment section, starts again with an overview of the different experiments, introducing their aspects and outlining how they are meant to analyse key aspects of the generative approach. 

> Additionally, the paper should include ablation studies to demonstrate the contribution of each MAGNet component, e.g., normalising flow, atom type on the shape level.

We are happy to include ablation studies as we already have provided the results for the atom type inclusion in our intial response. Regarding the normalising flow, we find the analysis on active units and insufficiency of the KL-regularisation more convincing. However, we have also analysed the generative performance of a MAGNet model without the normalising flow:
- We observed that the performance for the experiments from section 4.1 and 4.3 and 4.4 remained largely unchanged. 
- The unconditional generation, however, as analysed in 4.2 decreased to a score of 0.65 according to the FCD metric. 
 
As these analyses do not directly reinforce the primary arguments of the paper, we have included them in appendix section C.4.

Reviewer comment (Reproducibility):
I do not understand how MAGNet generates the shape multiset mentioned in section 2.2. Can you give me more details?

Author response:
> How do you generate the shape multiset?

- We train a 2-layer transformer decoder to generate the shape multiset. We use the decoder to learn the shape set in an autoregressive manner. We updated the "shape-level" paragraph in Section 2.2 and provide a more detailed explanation of the multiset generation in the Appendix B.2.

Reviewer comment (Theory):
Also in section 2.2, when inferring the shape connectivity, does MAGNet need to provide atom type? Why does the atom type exist on the shape-level step?

Author response:
> Why is an atom type present on the shape level?

- The requirement of the atom type on the shape's adjacency helps to ensure that the generated shape allocations can be joined on the atom level. Strictly speaking, this is not required and we updated the "Factorising the data distribution" paragraph in Section 2 accordingly. We analysed the effect of this design choice and provide the results for this experiment here. Including the atom type of the join improves the quality of generated molecules by 16% according to the FCD score. We explain the reason for this design choice more thoroughly in the "shape-level" paragraph of Section 2.2.

Reviewer comment (Reproducibility):
How does MAGNet extract fragments/shapes? Can you provide more examples in section 2.1?

Author response:
> How do we extract the shapes? Can you provide more examples?

- We clarified the discussion about the fragmentation and extraction of shapes in Section 2.1 of the manuscript. We further provide additional samples of the shape set in Appendix A, where we illustrate examples that we only discussed verbally in the initial manuscript.

Reviewer comment (Writing):
In section 2.3, how you apply a Normalizing Flow to the latent space is confusing.

Author response:
> How the Normalising flow is applied is confusing

- We observed that solely optimising the KL-term leads to over-pruning [1]. Thus, we apply a normalising flow to the latent space. To do so, we rely on Conditional Flow Matching [2, 3].  We have clarified this point at the end of Section 2.3 of the revised manuscript.

Reviewer comment (Writing):
The paper uses the term shape to describe the intermediate results which I personally think the term topology is more appropriate.

Author response:
> Terminology: shapes -> topology

- We apologise for the confusion caused by the terminology. After internal discussion, we decided to stick with the shape term in the current version of the manuscript. However, we provided a clarifying statement at the beginning of Section 2, "Factorising the data distribution", when defining shapes and refer to the concept of topology in various other parts other the discussion.

Reviewer comment (Experiments):
Insufficient Comparative Analysis: The paper falls short in providing a comparative analysis with diffusion-based models, which are crucial in the domain of molecule generation. This lack of comparison might lead to an incomplete evaluation of MAGNet, as readers are left without a clear understanding of how the proposed model stands against these advanced alternatives. A thorough comparison, highlighting the strengths, weaknesses, and performance differences, would significantly enhance the paper’s credibility and provide a more comprehensive assessment of MAGNet.

Author response:
> Comparison with diffusion models


- We initially decided against adding a paragraph on diffusion models as these primarily focus on molecule generation in 3D space [1,3,4], which is different from the focus of our work. However, we agree that it is beneficial for the reader to present the entire spectrum of molecule representations (SMILES, graphs, geometries) and have added in a paragraph in Section 3 (Related Work, "Molecule generation") of the updated manuscript.
- We highlight DiGress [2] specifically as it is a discrete diffusion model for general graphs that is applicable to 2D molecular graphs. However, similar to SMILES-LSTM, a comparison between the models is difficult due to its implicit learning of representations. DiGress, in its current form, is not applicable to the majority of our experimental evaluations and downstream applications, such reconstruction experiments in Section 4.1 and 4.3, scaffold/shape conditioning in Section 4.4, or the interpolation between molecules in Appendix C.
- However, DiGress works for unconditional generation and we will include it in the presented benchmark, see Section 4.2. After training for >2,5 days on one A100 GPUs (already 2x the compute resources of any other model), the model achieves an FCD score of 0.6 (MAGNet has 0.76). For the benchmark computation, we discarded all invalid molecules, which corresponds to about 26% of the generated samples. Note that sampling is 33x slower than MAGNet.

Reviewer comment (Writing):
Deviation from Established Terminology: The terminology used to describe MAGNet’s generation process, specifically the terms "sequential" and "Attribute-Atom-Object (AAO)," deviates from the established "Sequential" and "One-shot" terminology commonly used in the field. This inconsistency could potentially create confusion and hinder the paper’s accessibility to readers familiar with the standard terms. Adopting the widely accepted terminology would ensure clarity and maintain consistency across the literature.

Author response:
> Unusual terminology

- We acknowledge your comment regarding terminology and updated our manuscript to follow the survey from Zhu et al. [6], that is, we replace "AAO" with "OS". Importantly, "AAO" refered to "all-at-once" (Section 3) rather than "all-atom-object", which was motivated by the work of Yang et al. [5].

Reviewer comment (Novelty):
Limited Reference to Foundational Works: The paper does not adequately reference foundational works and comprehensive surveys in the field of molecule generation, which could limit the readers’ ability to place MAGNet within the broader context of the field. Including references to seminal works such as "A systematic survey on deep generative models for graph generation," and "A survey on deep graph generation: Methods and applications," would provide a richer background, enhancing the paper’s credibility and informative value.

Author response:
> Missing surveys citations

- We are aware of these works [5,6,7] and apologise for the oversight. To put MAGNet in a broader context, we updated the manuscript accordingly, see Section 3 "Related work" and Section 1 "Introduction".

Reviewer comment (Experiments):
Exclusion of Alternative Molecule Representations: The exclusive focus on a graph-based representation for molecules in the paper neglects the discussion of other popular representations like fingerprints. This exclusion limits the comprehensiveness of the paper, as readers are not provided with a comparison or rationale for the chosen representation. Discussing various molecule representations, their advantages, disadvantages, and applicability, would contribute to a more holistic view of the field and strengthen the paper’s content.

Author response:
> Discuss the variety of representations for molecules

- In the initial manuscript, we focused on the differentiation between SMILES and graphs for molecule generation. We added a clarifying sentence on using descriptors for molecule representation in Section 3 to address your comment. For this, we followed the survey from Du et al. [7].

Reviewer comment (Evaluation):
I am curious about the reason behind not being able to effectively train the VAE model (i.e., prior matching term), but have to fit another NF to the latent space. Would you please provide more analysis here?

Author response:
- It is important to note that we conducted an extensive investigation into the ineffectiveness of the KL-term. During our analysis of the latent space, we found that optimising the KL-term leads to over-pruning, meaning many latent factors fail to learn anything and become inactive [1]. Despite trying different methods, such as introducing dropouts or epitomes, we could not solve this optimisation issue satisfactorily. As a result of this analysis, we chose to fit a normalising flow post-hoc to the VAE which was trained with low KL regularsation. We agree that this clarification would be beneficial to include in the manuscript and have updated Section 2.3 accordingly, providing the analysis of active units in Appendix B.3 as well as an ablation study on this in Appendix C.4.

Reviewer comment (Evaluation):
Fig 3b, does the x-axis show percentage (e.g., 0.50%) or ratio? Also, fragment-based methods model the distribution of shape&atomic species. Therefore, it is reasonable that if you marginalize over the atomic species and only analyze the shape coverage, MAGNet will by design do better at diverse shape sampling.

Author response:
- Thank you for your feedback on Fig 3b's axis label. The figure had a typo, and we intended to refer to the ratio. As you correctly pointed out, MAGNet is designed to capture the structural variety of shapes. However, note that this design choice introduces additional complexity for generating shape allocations, which MAGNet captures to such an extent that it even outperforms other methods, highlighting its greater flexibility shown in Figure 4. We demonstrate MAGNet's ability to learn meaningful atom allocations both qualitatively (4a) and quantitatively (4b). To clarify this point, we updated Section 4.1 ("MAGNet reliably decodes shapes") of our manuscript.

Reviewer comment (Reproducibility):
Could you please provide more details on fitting normalizing flow for the latent space?

Author response:
- Regarding details on the normalising flow, we clarify its optimisation in the revised version of Section 2.3 and put it in context with the above explanations.

Reviewer comment (Experiments):
Consider a particular generation task where a specific motif is desired to be present in the generated molecule, can MAGNet be tailored for this task?

Author response:
- One can readily use MAGNet for this kind of conditional task. We have shown results for this task in Figure 5 (top) of the initial manuscript. As this might have been unclear before, we improved the experiment discussion in the corresponding paragraph of Section 4.4.