Paper: Conversational Drug Editing Using Retrieval and Domain Feedback (ICLR-2024)

Reviewer comment (Evaluation):
The domain feedback function in the ReDF is defined as the evaluation metric. I wonder if it may potentially cause 'information leakage', as it is accessing the information of 'success sequence editing'.

Author response:
**Q: Concerns about information leakage**

Thank you for raising this question. First, we would like to claim that from the data aspect, there is no information leakage. The question is indeed about the evaluation, and we agree that there could exist evaluation bias. However, coming up with an accurate and computationally feasible evaluation metric is still an open challenge in the AI for science research field. If the community has developed any better evaluation metrics, they can be incorporated into the ChatDrug pipeline smoothly. Besides in our work, to alleviate this issue, we have tried our best to include the qualitative analysis on all 39 tasks.

Reviewer comment (Writing):
The paper includes much domain knowledge for demonstration, which causes troublesome in comprehending the main idea. The authors may consider simplify the terms and focus on the main experimental phenomenon only.

Author response:
**Q: Simplify the terminologies and interpretation**

We apologize for any inconvenience our work may have caused you in your reading experience. When we wrote the paper, we tried to add visual explanations and qualitative analysis for all the terminologies and results, for example, Figure 1, Table 3, and Figures 2-4. We are aware that reaching a good balance between the simplicity-and-expertness trade-off is important, especially if we want to show the potential of our work to the ML community. Thus, if you can help list the specific sentences that confuse you, we are happy to answer them and improve the understandability of our work.

Reviewer comment (Experiments):
The paper does not provide a clear comparison or analysis of the different LLM backbones used in ChatDrug.

Author response:
**Q: Comparison and analysis of different LLM backbones**

Thank you for raising this question. For the main results, we have comparison results of three different LLMs as backbone: Galactica, Llama2, Turbo (ChatGPT) in the first version. Through the comparison, we find that the Turbo (ChatGPT) backbone can help ChatDrug achieve the best performance and highest stability for all three drug types. ChatDrug with GALACTICA as backbone only performs better than baseline on the small molecule editing and peptide editing but cannot deal with protein editing tasks. ChatDrug with Llama2 as backbone can achieve better performance than baseline methods on peptide editing and protein editing but fails to cover small molecule editing tasks. Such unstable performance across different drug types may be attributed to the limited domain specific training data for open-source LLMs like GALACTICA and Llama2. You can find the details in Table 1, 2, 4, 5.

Reviewer comment (Evaluation):
The paper does not provide any user study or evaluation from domain experts to validate the usefulness and usability of ChatDrug.

Author response:
**Q: Validation from domain experts.**

Thank you for asking this question. We completely agree that for works like ChatDrug, validation from the domain experts is definitely important. Thus, in our manuscript, we have done the following analysis:
- For small molecules (Table 3), we show how the molecule substructures change using ChatDrug. For example, in Table 3a, through conversational guidance, ChatDrug-Turbo changes the methyl group of the input molecule to an aminoethyl group, successfully obtaining a less soluble molecule.
- For peptides (Figure 2), we show how the motifs change using ChatDrug.  For instance, for task 301 (editing input peptide binding to HLA-B*44:02), the edited peptides can successfully upweight the alphabet E (glutamic acid) at position 2.
- For proteins (Figure 3), we show how the secondary structures change using ChatDrug. The optimized parts (alpha-helix and beta-sheet) are highlighted in blue circles.

Reviewer comment (Evaluation):
How do you ensure the quality and reliability of the retrieval and domain feedback module? How do you handle the cases where the retrieved information is inaccurate or outdated?

Author response:
**Q: Handling the inaccurate or outdated retrieved information**

Thank you for raising such an interesting question! We had the same concern when we wrote the paper, and that’s why we qualitatively show three molecules for each editing task in Table 3.

- As shown in Eq 2, the retrieved information is guaranteed to be accurate. However, this does not mean they always have a positive effect.
- In another way, your question can be translated to “Can ChatDrug handle cases where the retrieved information has a negative impact?” The answer is yes. In Table 3, for each editing task, we show three molecules: input (blue), intermediate and wrong molecules (red), and the final output molecules (green). The intermediate molecules do not satisfy our conditions, yet ChatDrug possesses the ability to correct this in an iterative manner.

Thank you for the in-depth discussion again.

Reviewer comment (Evaluation):
How do you measure the similarity between the input and output drugs? How do you balance the trade-off between similarity and diversity in drug editing?

Author response:
**Q: Concerns about trade-off between similarity and diversity**

This is another critical question. So for now, we mainly balance the similarity and diversity through prompt engineering. For example in the LLM, we added a phrase in the prompt, requesting “the output drugs should be similar to the input drugs” (Appendix F). For the ReDF evaluation function (Eq 2), we use Tanimoto similarity for small molecules and edit distance for peptides and proteins.

Reviewer comment (Experiments):
I am not sure about the small molecules experiment but at least for the other two, the only baselines provided are random. However, I don't feel this is fair since the author's approach is seeing a few positive examples. Would be great to have a baseline that uses a similar number of examples.

Author response:
**Q: More baselines for peptides and proteins.**

First thank you for raising this question. Our contents are a little dense, so we move some of our discussions into the appendix in the first version. We are very happy to explain more details to you, and hope the following answers can solve your questions.
- In Appendix I, we do provide the zero-shot ($C=0$) editing results on peptides and proteins. 
Editing performance of the zero-shot setting is as bad as the random mutation baseline. For example, the hit ratio of task 502 (more strand for protein editing) for zero-shot (23.73%) is only slightly higher than the random mutation (21.44%).
- However, one advantage of our ChatDrug pipeline is that we can keep improving the baselines in a few-shot manner, by utilizing the retrieval and conversation module. We can further improve the hit ratio to 59.68% through the ChatDrug pipeline for task 502. This is one of the main advantages of ChatDrug.
- On the other hand, the methods using positive and negative samples are called *supervised controllable generation* (you also mentioned relevant papers in the following, thanks again). But this is slightly different from the few-shot setting ($C$-shot) in ChatDrug, because it is hard to train a powerful classifier on less than 5 positive samples. Thus, we want to claim that such a few-shot editing is indeed one advantage of ChatDrug.
- We want to highlight that in ChatDrug, the main goal is to prove that such a novel paradigm (in-context and conversation) can lead to more accurate performance on drug design tasks.

Reviewer comment (Experiments):
The tasks focus on some basic properties of molecules like water solubility. However, often what we are really interested in drug design is a quantitative measurement like the binding affinity to a given target (or something similar). I don't see any results along these lines.

Author response:
**Q: Experiments of binding tasks.**

The problem is very important but challenging. The first challenge is about how to get an accurate textual description, since it may relate to the 3D structures of ligands and proteins. The next challenge is the dataset. For instance, if we take the protein-ligand (small molecule) binding from PDB, there are only a few ligands that can bind to the target proteins, i.e., we cannot construct a sufficiently large retrieval database for the few-shot setting in ChatDrug.

Reviewer comment (Experiments):
In small molecules there exist benchmarks that measure the binding to specific targets or other more detailed attributes. For example the datasets/baselines used in this paper: https://arxiv.org/pdf/2206.07632.pdf

Author response:
**Q: Related papers about molecular and protein supervised editing.**

Thank you for raising this concern, this is interesting for future exploration and is slightly different from our current focus. We will explain this in detail below.
-  The MOOD paper proposes the score-matching method with OOD control, where the control is a classifier pretrained on the whole ZINC dataset. In other words, this is the supervised editing task, which needs a classifier-guided generation. The key to success is how to train a powerful classifier.
- This also holds for the protein editing/optimization papers shared by you.
- However, in ChatDrug, we retrieve only $C$ small molecules/proteins after $C$ rounds of conversation. Our setting is of the few-shot ($C$-shot) editing.
- Last but not least, although these papers are in different settings, supervised editing (like MOOD) is definitely interesting to incorporate into ChatDrug. We added this in our revised manuscript (Conclusion section).

Reviewer comment (Experiments):
(Maybe less related since the paper's focus seems to be more on small molecules with proteins a secondary experiment) For proteins there exist benchmarks like Deep Mutational Scanning and FLIP and associated works that try to optimize a protein towards one of these attributes. https://www.biorxiv.org/content/10.1101/2021.11.09.467890v1 https://www.biorxiv.org/content/10.1101/2021.07.09.450648v2 https://arxiv.org/abs/2303.04562 https://arxiv.org/abs/2307.00494

Author response:
Hi Reviewer pPSW,

Thank you for the clarification! We definitely agree that these tasks can be adapted into our ChatDrug pipeline, using the training set as the retrieval DB. Thank you again for sharing them, and we are happy to include them in our next project.

One thing we may kindly need your help with is boosting the score, as we've noticed it remains unchanged.

Thank you again for the positive feedback!

Regards, Authors of ChatDrug

Reviewer comment (Experiments):
I am curious as to what baselines the authors think are fair for each task that use similar amounts of supervised data. This is a bit unclear to me in the paper.

Author response:
**Q: Fair baselines in LLM setting.**

It is challenging for us to add baselines in the first place, because using LLM for few-shot editing is a very novel paradigm, and we are actually creating a new research line. Thus, we would claim that the most fair comparison would be to use ChatDrug with different LLMs for the same task setting. This is also the reason why we introduced three types of backbone LLMs (Galactica, Llama2, and Turbo/ChatGPT) in the ChatDrug pipeline.