Paper: Text-guided Diffusion Model for 3D Molecule Generation (ICLR-2024)

Reviewer comment (Novelty):
A similar idea has been proposed and published already, called SILVR (https://pubs.acs.org/doi/epdf/10.1021/acs.jcim.3c00667). In contrast to SILVR, the authors propose to create a reference geometry first using an ML-based mapping from textual description to geometry. Hence the generative model is not guided directly by text, but by some reference geometry the text-geometry mapping is outputting.

Author response:
**Comparison with IVLR (W1, W5)**: We would like to clarify that our work fundamentally differs from IVLR. While both methods aim at generating 3D molecules, we propose a fine-tuning approach using reference geometry for a diffusion model, whereas ILVR is a sampling method that does not require training.

Reviewer comment (Evaluation):
QM9 contains only very small molecules. A better benchmark set would be something like GEOM-Drugs. Results are missing error bars. In particular, the results in Table 2 are only marginally better and probably not significant.

Author response:
**Evaluation Concerns (W6, Q8, W3, W4)**: The evaluation method we used for assessing the quality of the generated molecules with respect to their conditioned property is a common approach adopted by mainstream methods such as EGNN and EEGSDE. It focuses on the quality of the generated molecules in terms of their conditioned properties rather than the quality of the generated 3D molecular structures. We used the same random splitting method of the dataset and trained the network $\phi$, so the answer for Q7 is no. As for the GEOM-Drugs dataset, its complexity makes it challenging to design specific conditional generation experiments; therefore, we did not include it in our study. We acknowledge that error bars are missing due to dense data tables and numerous experiments. However, improvements in Novelty and Stability are indeed challenging aspects in diffusion-based 3D molecular generation.

Reviewer comment (Evaluation):
The results on PubChem have only been evaluated on selected examples. The reader is left guessing whether the text aligns with the depicted molecules; no additional metrics are given to prove correlation / correctness.

Author response:
**Evaluation Concerns (W6, Q8, W3, W4)**: The evaluation method we used for assessing the quality of the generated molecules with respect to their conditioned property is a common approach adopted by mainstream methods such as EGNN and EEGSDE. It focuses on the quality of the generated molecules in terms of their conditioned properties rather than the quality of the generated 3D molecular structures. We used the same random splitting method of the dataset and trained the network $\phi$, so the answer for Q7 is no. As for the GEOM-Drugs dataset, its complexity makes it challenging to design specific conditional generation experiments; therefore, we did not include it in our study. We acknowledge that error bars are missing due to dense data tables and numerous experiments. However, improvements in Novelty and Stability are indeed challenging aspects in diffusion-based 3D molecular generation.

Reviewer comment (Experiments):
Ablation studies are missing: Is the improved performance in Table 1 compared to EDM due to text description, or because ILVR is the better conditioning approach?

Author response:
**Comparison with IVLR (W1, W5)**: We would like to clarify that our work fundamentally differs from IVLR. While both methods aim at generating 3D molecules, we propose a fine-tuning approach using reference geometry for a diffusion model, whereas ILVR is a sampling method that does not require training.

Reviewer comment (Evaluation):
The evaluation in Table 1 is only done with a property network instead of actual ground truth calculated values. Beyond that they do not seem to be of relaxed structures and the quality of the generated geometries is not checked. Therefore, the classifier might be out of training domain and/or the structure-dependent property might be quite different in equilibrium compared to the predicted 3d structure.

Author response:
**Evaluation Concerns (W6, Q8, W3, W4)**: The evaluation method we used for assessing the quality of the generated molecules with respect to their conditioned property is a common approach adopted by mainstream methods such as EGNN and EEGSDE. It focuses on the quality of the generated molecules in terms of their conditioned properties rather than the quality of the generated 3D molecular structures. We used the same random splitting method of the dataset and trained the network $\phi$, so the answer for Q7 is no. As for the GEOM-Drugs dataset, its complexity makes it challenging to design specific conditional generation experiments; therefore, we did not include it in our study. We acknowledge that error bars are missing due to dense data tables and numerous experiments. However, improvements in Novelty and Stability are indeed challenging aspects in diffusion-based 3D molecular generation.

Reviewer comment (Writing):
The text contains many spelling mistakes and unscientific language, the captions of plots and tables are often insufficient, citations are partly wrong.

Author response:
**Other Considerations (W7, Q6, Q7)**: Regarding your question about matching $c_\text{P}$ with atoms generated by the diffusion model (Q6), we don't require an exact match in terms of atom count – for cases with fewer atoms, we apply zero-padding for non-corresponding atoms. We have rectified spelling mistakes and unscientific language in our manuscript; a revised version will be uploaded shortly.

Reviewer comment (Novelty):
Why is it important to generated 3d molecules? It seems for the text-to-molecule use case generation of molecular graphs would be preferable.

Author response:
**Importance of 3D Molecule Generation (Q1)**: As discussed in our paper, generating 3D molecular structures provides more detailed and precise information about the molecule's spatial arrangement and its potential chemical properties compared to 2D molecular graphs. This is particularly useful in numerous applications such as drug discovery and materials science.

Reviewer comment (Theory):
Why should conditioning on text work better than on numerical values, particularly for single property conditioning? What additional information does the text description contain here?

Author response:
**Advantages of Textual Conditioning over Numerical Conditioning (Q2, Q3, Q5)**: The superiority of textual conditioning over numerical conditioning lies not in the additional information provided by specific numerical values but in its ability to capture intricate conditions that may not be sufficiently represented by a single or a handful of target properties. For instance, "small HOMO-LUMO gap" as mentioned in Fig 1 is an example where the text provides a more nuanced condition than could be captured by numeric values alone.

Reviewer comment (Reproducibility):
How is the reference geometry c_p obtained? What is the precise architecture of the “multi-modal conversion module”? How is it trained?

Author response:
**Description of Multi-modal Conversion Module & Reference Geometry (W2, Q4, Q5)**: Our multi-modal conversion module $\Gamma$ comprises a GIN molecular graph encoder and a language encoder-decoder extended from BERT which we pretrained on a randomly selected subset of our PubChem dataset for converting textual prompts to reference geometry $c_\text{P}$. We believe this approach does not restrict the conditioned diffusion model as we do not aim for deterministic output given certain conditions; instead, we strive for exploring the molecular space which aligns with our motivation.

Reviewer comment (Theory):
The text-to-reference geometry encoding cannot be a deterministic mapping, as multiple geometries could correspond to the same text description. Why should this not restrict the conditioned diffusion model? Why is the “reference geometry” in Fig 2 not a connected molecule?

Author response:
**Description of Multi-modal Conversion Module & Reference Geometry (W2, Q4, Q5)**: Our multi-modal conversion module $\Gamma$ comprises a GIN molecular graph encoder and a language encoder-decoder extended from BERT which we pretrained on a randomly selected subset of our PubChem dataset for converting textual prompts to reference geometry $c_\text{P}$. We believe this approach does not restrict the conditioned diffusion model as we do not aim for deterministic output given certain conditions; instead, we strive for exploring the molecular space which aligns with our motivation.

Reviewer comment (Reproducibility):
If c_p is a 3d structure, how is it matched with the atoms generated by the diffusion model? Has it necessarily the same number of atoms?

Author response:
**Other Considerations (W7, Q6, Q7)**: Regarding your question about matching $c_\text{P}$ with atoms generated by the diffusion model (Q6), we don't require an exact match in terms of atom count – for cases with fewer atoms, we apply zero-padding for non-corresponding atoms. We have rectified spelling mistakes and unscientific language in our manuscript; a revised version will be uploaded shortly.

Reviewer comment (Reproducibility):
Since the text dataset used in Section 4 is not publicly available, it is hard to reproduce their results in the subsequent research.

Author response:
We understand your concern about the reproducibility of our results due to the unavailability of the text dataset used in Section 4. We have prepared the dataset for open-source release. However, in line with the double-blind review process, we will fully release the dataset upon the official publication of the paper. We believe this will ensure that subsequent research can reproduce and build upon our results. Furthermore, we are continuously maintaining this dataset from the PubChem database which is also constantly updated. This will ensure that the dataset remains relevant and useful for future research.

Reviewer comment (Experiments):
Although the direct use of C_p is not recommended in the main text, the empirical evaluation of it is not available. Since the proposed method is complex, the readers would want to see more supporting evidence of the current design choice.

Author response:
We apologize for any confusion caused, and we appreciate your feedback. As we mentioned in the main text, valid and stable 3D molecules can hardly be obtained directly from $c_P$, which makes it difficult to evaluate it with a metric suitable for molecular generation. We ubderstand your concerns regarding the complexity of our novel approach of incorporating text guidance in diffusion-based 3D molecular generation. To substantiate the efficacy of our model design, we have performed an array of experiments, detailed in the experimental section. We concur that refining and simplifying the model structure are worthwhile pursuits, and we are diligently working towards this end. It is our hope to furnish more corroborative evidence in our forthcoming work.

Reviewer comment (Writing):
Is the reference of EEGSDE to (Igashov et al., 2022) correct? Do you mean Equivariant Energy-Guided SDE for Inverse Molecular Design, https://arxiv.org/abs/2209.15408?

Author response:
We appreciate your attention to detail. Indeed, there was an error in our reference to EEGSDE. It was incorrectly cited as (Igashov et al., 2022). We have corrected this in our revised paper and the reference now points to the correct source. We apologize for this oversight and thank you for bringing it to our attention. The revised paper will be uploaded shortly, and we kindly request you to review it at your convenience.

Reviewer comment (Experiments):
The dataset is constructed manually with a set of textual templates, and the authors do not give details on how to build it. This may lack diversity and be far away from real-world challenges.

Author response:
**Q1**: We appreciate your interest in our dataset. We have curated a text-molecule dataset from PubChem, one of the most comprehensive databases for molecular descriptions. It collects annotations from various sources including ChEBI, LOTUS, and T3DB, each focusing on different molecular properties. Our dataset comprises 330K text-molecule pairs, which we believe sufficiently represent a wide diversity of molecular structure and properties. The dataset was assembled as follows:
1. We downloaded textual molecule descriptions using PUG View, a REST-style web service provided by PubChem. This yielded over 382,000 annotations.
2. The downloaded record descriptions were sorted according to the isomeric SMILES of molecules, resulting in 330,212 unique molecules with descriptions from multiple data sources within the PubChem database.
3. To standardize the dataset, we replaced IUPAC names or synonyms at the beginning of each description with "The molecule ...".
4. Finally, we used RDKit to convert the SMILES into graph-structure data paired with their textual descriptions.

Reviewer comment (Evaluation):
The experiment in Tab. 3 is conditioned on two quantum properties. However, the results only report the MAE of a single property, lacking the proportion of molecules that meet both conditions.

Author response:
**Q3**: We adopted the experimental approach from EEGSDE and compared our method with it under multiple conditions in Table 3. Due to the different numerical values and scales of various properties, it's challenging to evaluate multiple numerical properties simultaneously. Moreover, due to the computational cost of training diffusion models, reproducing all baselines on a new metric is difficult. We acknowledge this limitation and plan to address it in future work.

Reviewer comment (Experiments):
Can you provide more details about the dataset you collected and analyze if it lacks diversity?

Author response:
**Q1**: We appreciate your interest in our dataset. We have curated a text-molecule dataset from PubChem, one of the most comprehensive databases for molecular descriptions. It collects annotations from various sources including ChEBI, LOTUS, and T3DB, each focusing on different molecular properties. Our dataset comprises 330K text-molecule pairs, which we believe sufficiently represent a wide diversity of molecular structure and properties. The dataset was assembled as follows:
1. We downloaded textual molecule descriptions using PUG View, a REST-style web service provided by PubChem. This yielded over 382,000 annotations.
2. The downloaded record descriptions were sorted according to the isomeric SMILES of molecules, resulting in 330,212 unique molecules with descriptions from multiple data sources within the PubChem database.
3. To standardize the dataset, we replaced IUPAC names or synonyms at the beginning of each description with "The molecule ...".
4. Finally, we used RDKit to convert the SMILES into graph-structure data paired with their textual descriptions.

Reviewer comment (Reproducibility):
What are the pre-trained data and training objectives of the multi-modal module? Why is the geometry information contained in the condition vector?

Author response:
**Q2**: For pre-training our multi-modal module, we utilized 300K text-molecule pairs from the Pubchem dataset following a methodology similar to CLIP's architecture. The Graph Encoder is based on GIN and KV-PLM (based on BERT) is used as the text encoder. The alignment between graph representation and text representation is achieved via InfoNCE loss. Subsequently, we trained a multi-modal decoder using over 10K text-molecule pairs to generate molecular graph structures based on corresponding texts. For generating reference geometry information, we identified functional groups using RDKit toolkit and created 3D conformers accordingly.

Reviewer comment (Experiments):
Why not compare it with the previous baseline and report the metrics about whether your generated molecules meet multiple conditions at the same time?

Author response:
**Q3**: We adopted the experimental approach from EEGSDE and compared our method with it under multiple conditions in Table 3. Due to the different numerical values and scales of various properties, it's challenging to evaluate multiple numerical properties simultaneously. Moreover, due to the computational cost of training diffusion models, reproducing all baselines on a new metric is difficult. We acknowledge this limitation and plan to address it in future work.

Reviewer comment (Reproducibility):
Many existing diffusion-based models for generating molecules (or point clouds more generally) have a caveat around number of atoms (points), which has to be fixed beforehand. Is it also the case here? When sampling from the model, do you sample the number of atoms separately? Is that conditioned on the text?

Author response:
**(a)** In our model, we do not sample the number of atoms separately. For cases where the number of atoms is relatively small, we resort to padding with a value of 0, which does not correspond to any atom.

Reviewer comment (Reproducibility):
How does $\Gamma$ work? As I understand, it is a model mapping from text to a molecular conformation, i.e. the output is a set. How is this set decoded? Are there any issues in pretraining this component related to the fact that the atoms do not have a predefined ordering?

Author response:
**(b)** Our multi-modal module is pretrained using 300,000 text-molecule pairs from the PubChem dataset. Specifically, we adopt an architecture akin to CLIP, utilizing GIN as our Graph Encoder and KV-PLM (based on BERT) as our text encoders. We employ $L$ transformer layers equipped with self-attention (SA) and feed-forward network (FFN) blocks to generate hidden states and pool them into representations. Subsequently, we align the graph representation and text representation using an InfoNCE loss function. After this step, we train a multi-modal decoder on more than 10,000 text-molecule pairs to generate molecular graphs based on corresponding texts. Lastly, by leveraging RDKit's functionality for identifying functional groups and generating 3D conformers, we create a reference geometry.

Reviewer comment (Reproducibility):
What are the different training stages involved here? I understand $\Gamma$ is trained beforehand and frozen; is the unconditional EDM also pretrained (without involving text) and frozen? Are the parameters of the linear transformation $\theta$ then trained separately, in a third stage? I can't fully match this with Figure 2 which suggests two stages, not three.

Author response:
**(c)&(d)** We apologize for any confusion caused by Figure 2 --- due to computational constraints, we utilized a pretrained EDM model (not involving text), which was not depicted in Figure 2's illustration of training stages. During training, only $\theta$ is further trained. It's crucial to note that our objective was to propose a framework for a text-guided diffusion model for 3D molecule generation; in future work, we aim for more comprehensive training leveraging richer datasets and computational resources. Indeed, the number of text-molecule pairs (330K) that we collected from PubChem is relatively limited and only a portion of these data can yield corresponding 3D conformations.

Reviewer comment (Evaluation):
Table 2 presents some results, bolding the best numbers, however, the differences between most of these results are tiny, and they don't look statistically significant. Some significance test would be good here. Could the authors also discuss why the differences are so small? It seems the novelty/stability percentages are influenced much more strongly by the property being conditioned on than the generative model itself, which is somewhat surprising and counter-intuitive.

Author response:
Concerning Table 2, we concur that the differences among results are minor and could benefit from a significance test. In fact, it is reasonable to expect some variability in the novelty displayed by the diffusion model under different conditions. However, stability, calculated through atomic valence, inherently exhibits considerable fluctuations in experiments. We adhered to the settings of prior work when reporting experimental results and aspire to propose more persuasive evaluation methods in future work.