Paper: Data-Efficient Molecular Generation with Hierarchical Textual Inversion (ICLR-2024)

Reviewer comment (Novelty):
It is worth mentioning some recent works on molecule generation in related works such as:

[1] Hoogeboom E, Satorras V G, Vignac C, et al. Equivariant diffusion for molecule generation in 3d, ICML 2022  
[2] Zhang Z, Liu Q, Lee C K, et al. An equivariant generative framework for molecular graph-structure Co-design. Chemical Science 2023  
[3] Flam-Shepherd D, Zhu K, Aspuru-Guzik A. Language models can learn complex molecular distributions. Nature Communications, 2022

Author response:
We appreciate your comment to improve our manuscript. Following your suggestion, we mentioned the related works [1, 2, 3] in Section 2 of our revised manuscript.

Reviewer comment (Experiments):
Could HI-Mol leverage the structural information of molecules? Could it be adapted for 3D molecule generation?

Author response:
We strongly believe that our framework can be adapted for 3D molecular generation by leveraging the 3D structural information of molecules. Although we use the state-of-the-art text-to-molecule model (MolT5), which utilizes the SMILES string as the molecule representation, it is certainly possible to model the 'molecule' part of a text-to-molecule model as 3D molecule (with a recently proposed 3D molecular generative model [1] as a decoder architecture) without any methodological modifications of our method.

Reviewer comment (Evaluation):
How to effectively interpret the learned tokens? Do they have chemical meanings?

Author response:
Learned tokens have a chemical meaning (i.e., hierarchical features of molecules). For example, detail tokens represent features of a single molecule in the dataset. These learned tokens are effectively interpreted through visualization. For example, in Figure 2 of the manuscript, one can see that certain intermediate tokens represent molecules with chemical substructures such as long carbon chains or sulfonyl groups.

Reviewer comment (Presentation):
The authors should have a clearer motivation figure in the introduction, which could be specific examples of molecules, to demonstrate that the highly complicated and structured nature of molecules makes it difficult to apply textual inversion directly.

Author response:
**[W1] Clear motivation figure to demonstrate highly complicated and structured nature of molecules.**

Thank you for the comment to improve the clarity of our manuscript. Following your suggestion, we added Figure 2 as a motivation figure in the introduction section of our revised manuscript.

Reviewer comment (Experiments):
Table 2 should also show the results of HI-Mol without grammar.

Author response:
**[W3] Results of HI-Mol without grammar.**

Thank you for the suggestion. We moved the QM9 results of HI-Mol without grammar to Table 3 (which is originally placed in Table 15 of Appendix G), in our revised manuscript.

Reviewer comment (Evaluation):
In Table 6, Valid decreases as the token hierarchical levels increases. The authors should provide some explanations and solutions for this issue.

Author response:
**[W4] In Table 6, why Valid decreases as the token hierarchical level increases? How to resolve it?**

As the token hierarchy increases, Validity decreases because the model interpolates more levels of embeddings to generate more diverse molecules (measured by Unique. and Novelty), i.e., there is a trade-off between the diversity of generated molecules and their validity. However, we emphasize that an invalid molecule can be easily converted to a valid molecule via a simple modification algorithm (in Appendix H), which actually follows our desired distribution (see improvement in FCD score in the table below).

\begin{array}{l|c|ccccc}
\hline
\text{QM9} &\text{Grammar} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline
\hline
\text{HI-Mol (Ours)} & \text{X} &  0.434 & 0.001 & 90.7 & 75.8 & 73.5 \newline
\text{HI-Mol (Ours)} & \text{O} &  \textbf{0.430} & \textbf{0.001} & \textbf{100} & \textbf{76.1} & \textbf{75.6} \newline
\hline
\end{array}

Reviewer comment (Presentation):
I suggest that the authors consider rearranging the positions of the figures and tables as some of them are too far from the references in the paper.

Author response:
**[W5] Rearrangement of the positions of the figures and tables.**

Thank you for the comment to improve our manuscript. Following your suggestion, we carefully rearranged the positions of the figures and tables in our revised manuscript.

Reviewer comment (Reproducibility):
What is the ‘simple resampling strategy’ mentioned in Section 4.2?

Author response:
**[Q1] What is ‘simple resampling strategy’ in Section 4.2?**

Thank you for the opportunity to clarify this point. The ‘simple resampling strategy’ means that we ignore the generated sample when it is (1) invalid, (2) identical to an already generated sample, or (3) identical to a sample in the training dataset. By applying this strategy, we obtain a set of generated samples whose Valid., Unique., and Novelty scores are all 100 (note that sampling costs only 1.8 sec per molecule). We consider this scenario to align the Valid., Unique., and Novelty scores with some methods (e.g., GDSS) for a fair comparison in the FCD score. Even in this scenario, HI-Mol significantly outperforms those methods.

Reviewer comment (Reproducibility):
The method is not described clearly and in detail. For instance, in the following paragraph of Eq. 1, it is mentioned that the intermediate tokens are "selected" during training. This is unclear and should be discussed in more detail.

Author response:
**[W1] Clear description of method, e.g., “selected” in Eq. (1).**

Thank you for the comment to improve the quality of our manuscript. For clarification, to learn the cluster-wise features of a molecular dataset, we propose a strategy to assign an appropriate cluster index to each molecule. Specifically, given a molecule, we assign (i.e., “select’’) the intermediate token $[I_k]$ where $k$ minimizes the reconstruction loss among the cluster indices $[1, K]$. This strategy allows us to cluster similar molecules (see Figure 2) in an unsupervised manner. We carefully revised the manuscript to provide a clear description of our method.

Reviewer comment (Presentation):
Figure 1 is not expressive enough to outline the method.

Author response:
**[W2] Figure 1 is not expressive enough to outline the method.**

Thank you for the suggestion. Following your comment, we revised Figure 1 to better outline our method.

Reviewer comment (Experiments):
The authors have mentioned that they are using the Caption2Smiles frozen model as the backbone. Can they please share just the frozen model performance on the tasks?

Author response:
**[Q1] Performance of the frozen Caption2Smiles model?**

Following your suggestion, we provide the generation results of the frozen Caption2Smiles model [1] on the HIV dataset, where we utilize the task description of the HIV dataset as the text prompt. The table below shows that our method indeed significantly outperforms naïvely prompting frozen Caption2Smiles model, which implies that our proposed framework is essential in achieving the superior performance. We added these respects of discussion and the experimental details in Appendix F of our revised manuscript.

\begin{array}{l|cccccc}
\hline
\text{HIV} & \text{Active. $\uparrow$} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline
\hline
\text{MolT5-Large-Caption2Smiles [1]} & 0.0 & 60.6 & 0.196 & \textbf{100} & 14.6 & \textbf{100} \newline
\hline
\textbf{HI-Mol (Ours)} & \textbf{11.4} & \textbf{16.6} & \textbf{0.019} & \textbf{100} & \textbf{95.6} & \textbf{100} \newline
\hline
\end{array}

Reviewer comment (Evaluation):
Sensitivity of the model's performance to the number of k sounds very important. According to Appendix E, table 10, there is no benefit in increasing k to more than 10. Do the authors have a hypothesis for that? Aren't the molecules supposed to have a lot more "clusters", sub-components?

Author response:
**[Q2] In Table 10, there is no benefit in increasing $K$ more than 10. Do the authors have a hypothesis?**

Thank you for the insightful comment. We kindly remark that some previous works on molecular clustering have shown that a thousand of molecules can fall into $\sim$10 clusters [2], which could be a supporting hypothesis on our finding $K=$ 10. While the number of “clusters’’ of molecules can be defined in various ways, we also hypothesize that choosing a relatively small number of clusters (e.g., $K=$ 10) is sufficient in our framework. This is because the sharing of the coarse-grained common features (i.e., intermediate tokens) serves to regularize the fine-grained features (i.e., detail tokens) which are biased toward a single molecule in the embedding interpolation-based sampling. To verify our hypothesis, we conduct an additional experiment, where we replace the intermediate tokens with additional detail tokens for each molecule (i.e., $K=$ 2,113). The reported result below show that the overall performance is rather degraded with $K=$ 2,113 compared to $K=$ 10, 20, and 30. This verifies our hypothesis and justifies our choice of the number of clusters $K$ in our experiments. We added these respects of discussion in Appendix E of our revised manuscript.

\begin{array}{l|ccccc}
\hline
\text{$K$} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline
\hline
10 & 0.434 & \textbf{0.001} & \textbf{90.7} & 75.8 & 73.5 \newline
20 & \textbf{0.430} & \textbf{0.001} & 87.9 &\textbf{77.3} & 73.8 \newline
30 & 0.436 & \textbf{0.001} & 88.9 & 77.2 & \textbf{73.9} \newline
\text{2,113} & 0.443 & \textbf{0.001} & 86.2 & 75.4 & 72.6 \newline
\hline
\end{array}

Reviewer comment (Experiments):
Can the authors please provide the complete Qm9 results in table 15 of Appendix G? Specifically, what are the results at the 50% and 100% ratios?

Author response:
**[Q4] In Table 15, what are the results at the 50% and 100% ratio?**

Thank you for the suggestion to improve our manuscript. First, we would like to emphasize that our target problem is molecular generation in the low-data regime, and our method already outperforms state-of-the-art methods with only 2% of training data of the QM9 dataset. For this reason, we thought that the results using 50% and 100% training data are not that meaningful to show the superiority of our method. Nevertheless, for your information, we are currently running the experiments with 50% and 100% QM9 dataset. We will definitely include these results in our final draft (we hope your understanding that it is costly to train with such a large number of molecules in our local machine).

Reviewer comment (Novelty):
The main concern I have with this paper is its novelty. While the ideas of multi-level molecule representation and embedding interpolation are well-established in the field, the authors merely integrate them into the newly introduced textual inversion framework. This casts doubts over the paper's genuine novelty and the depth of its technical contribution.

Author response:
We politely disagree with the reviewer’s concern; although the high-level concepts of multi-level representation and embedding interpolation have been used for various purposes and domains in the literature, their usage for our own purpose could be novel and not straightforward. Specifically, we strongly believe that our novelty lies in (1) introducing the textual inversion for molecular generation and (2) advancing the existing textual inversion technique (which originally uses a single shared token in the image domain) by introducing multi-level tokens. Here, (2) is motivated by our new finding that a naïve adoption of textual inversion using a single shared token does not perform well in molecular generation. To alleviate this issue, we introduce a new concept, namely multi-level tokens, into the textual inversion framework to capture the complex distribution of molecules. Furthermore, we carefully design the sampling strategy based on the interpolation, since it is not straightforward to sample molecules from our newly proposed multi-level token embeddings. Therefore, our method is not a straightforward integration of previous techniques, but a novel framework for successfully adapting the textual inversion to molecular generation by introducing multi-level tokens into the textual inversion framework.

Reviewer comment (Theory):
The rationale for adopting the textual inversion model appears somewhat nebulous. In my understanding, compared to SMILES, graph representations are generally more adept at modeling molecular structures. Notably, many graph-centric molecular generation methods have already incorporated hierarchical concepts. In their experiments, while the authors argue that HI-Mol surpasses several graph-based models, the underlying reasons remain unelucidated. An elucidation on how the textual inversion model specifically augments the molecular generation task, relative to its graph-based counterparts, would be greatly beneficial.

Author response:
For clarification, we use SMILES strings as the representation of molecules because our method is built upon the state-of-the-art text-to-molecule model, MolT5 [1], that utilizes SMILES strings, not graphs. On the other hand, if there exist graph-based text-to-molecule models, our method is also applicable to them (then, we would use graphs as the representation of molecules). Namely, our method is agnostic to the underlying molecule representation of the foundation models, and we do not argue for the superiority of the molecule representation between SMILES strings and molecular graphs.

Reviewer comment (Experiments):
There's an absence of crucial baselines. The authors have chosen to compare their work with two SMILES-based baselines, one based on RNN and the other on spanning tree. However, they have not included any methods based on large-scale text-to-molecule models, many of which are mentioned in the related works section. Therefore, it is unclear whether the improved performance of HI-Mol is attributable to the utilization of large-scale text-to-molecule models.

Author response:
Although our framework is built upon large-scale text-to-molecule models, they are not meaningful baselines. This is because it is not feasible to prompt all the training molecules to the mentioned large-scale text-to-molecule models due to the maximum token length of the input prompt. Specifically, while the maximum token length of the text-to-molecule model is only 1024 and 512 for [1] and [2], respectively, we have thousands of molecules and each SMILES representation of a molecule can sometimes consist of more than 100 tokens. Nevertheless, one can utilize the mentioned text-to-molecule models [1, 2] by providing the task description only (without providing the training molecules). For your information, the table below shows that our method indeed significantly outperforms such a naïve prompting of the text-to-molecule models on the HIV dataset. This confirms that our improved performance is not just due to the utilization of the large-scale text-to-molecule model, but to our carefully designed textual inversion framework. We added these respects of discussion and the experimental details in Appendix F of our revised manuscript.

\begin{array}{l|cccccc}
\hline
\text{HIV} & \text{Active. $\uparrow$} & \text{FCD $\downarrow$} & \text{NSPDK $\downarrow$} & \text{Valid. $\uparrow$} & \text{Unique. $\uparrow$} & \text{Novelty $\uparrow$} \newline
\hline
\text{MolT5-Large-Caption2Smiles [1]} & 0.0 & 60.6 & 0.196 & \textbf{100} & 14.6 & \textbf{100} \newline
\text{Text+Chem T5-augm-base [2]} & 0.0 & 62.2 & 0.188 & \textbf{100} & 90.2 & \textbf{100} \newline
\hline
\textbf{HI-Mol (Ours)} & \textbf{11.4} & \textbf{16.6} & \textbf{0.019} & \textbf{100} & \textbf{95.6} & \textbf{100} \newline
\hline
\end{array}


---
[1] Edwards et al., Translation between Molecules and Natural Language, EMNLP 2022\
[2] Christofidellis et al., Unifying Molecular and Textual Representations via Multi-task Language Modeling, ICML 2023