Paper: Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
While the performance improvements observed when fine-tuning the LLaMa model on the collected data are promising, this setup resembles a supervised learning scenario. Consequently, the comprehensiveness of the performance assessment may be somewhat limited. To provide a more comprehensive evaluation, additional comparisons with domain-specific Large Language Models (LLMs), such as those mentioned in references [1] and [2], would be valuable. Additionally, incorporating more challenging datasets such as ScienceQA [3] could further strengthen the paper's empirical results.

Author response:
**2. More domain-specific baselines**

In response, we have incorporated two new domain-specific Large Language Models (LLMs) into our evaluation: Galactica-6.7B[1] and PMC_LLaMA_13B[2]. This is because BioMedLM[3] and PMC_LLAMA_7B[4] are primarily suited for text-generation tasks. When prompted with questions and options, they tend to generate more options rather than providing answers, due to their lack of instruction tuning. To address this, we included PMC_LLaMA_13B, an instruction-tuned version of PMC_LLAMA_7B, as a new baseline in our study. 

The updated results reflecting these expansions can be found in **Tables 3 and 4**, as well as **Figures 5, 6, 8, 9, 10, and 11** in our revised manuscript. 

**3. More challenging datasets**

Regarding the inclusion of the ScienceQA[5] dataset, we note that the majority of ScienceQA's biology-related content is multimodal, which does not align well with the current focus of Mol-instructions on text and molecular/protein sequences. 

Incorporating multimodal data from ScienceQA would require a substantial extension of the Mol-instructions dataset to support formats beyond text and sequences. While this falls outside the scope of our current research, we recognize the potential value of such an expansion. It represents a promising direction for future work, where Mol-instructions could be enhanced to support a broader range of data types, including multimodal data. This would allow for a more comprehensive evaluation of LLMs in biomolecular research and is a pathway we are keen to explore in our future endeavors **(Highlighted in Section 6, Page 9)**.

Reviewer comment (Evaluation):
The paper would benefit from a more in-depth analysis of the factuality of the generated text.

Author response:
**4. Factuality analysis**

Thank you for highlighting the importance of a more in-depth analysis of the factuality of the generated text. The main challenge we encounter is the lack of a unified and widely accepted standard for evaluating tasks related to molecular/protein understanding. Consequently, we have relied on measurement methods used in previous NLP research.

We also attempted to use GPT-4 with certain predefined rules for scoring, but this approach did not yield stable scores. On the other hand, expert review, while potentially more accurate, is extremely time-consuming. Acknowledging these limitations, we are actively exploring more feasible and effective methods to analyze the factuality of the generated text, aiming to improve the robustness and reliability of our evaluation process.

Reviewer comment (Evaluation):
The evaluation may be a small issue; however, I know it is not easy to evaluate the molecule generation tasks.

Author response:
Thank you very much for your valuable suggestion. Below are our responses:

**Evaluation methods**

Indeed, we have made several attempts and efforts in this regard. For instance, we customized rules to enable GPT-4 to score molecules. However, the outcomes were not as satisfactory as we hoped. GPT-4 consistently produced unstable scores, failing to accurately assess the quality of the molecules. Alternative methods like wet-lab experiments or expert reviews, while potentially more accurate, are prohibitively expensive and time-consuming. We are actively seeking more feasible and efficient solutions to address this evaluation challenge.

Thank you again for your insightful feedback!

Reviewer comment (Novelty):
Recent work [1] showed that a large number of instructions may not be necessary to obtain a well-finetuned model.

Author response:
Thank you very much for your insightful comments. Below are our responses:

**Impact of Reduced Data on LLM Performance**

The main premise of LIMA[1] is that almost all knowledge is acquired during the pre-training phase, and only a limited amount of instruction tuning data is needed to guide LLMs in generating high-quality outputs. For the domain-specific data encompassed in Mol-instructions, LLMs may not have encountered such data during the pre-training stage. Therefore, using fewer instruction data might not adequately equip the model to grasp the response format of this particular dataset. Our goal in constructing Mol-instructions was to comprehensively cover various biomolecular tasks, thereby infusing the model with relevant knowledge. This ensures that even users focusing on specific tasks can effectively utilize this dataset for subsequent research, benefiting the entire community. 
Indeed, the instruction tuning process may not necessarily require the full set of instruction data. I believe your suggestion provides an excellent starting point for future work, which could focus on discussing the most effective selection methods for sampling the most beneficial alignment data.

Thank you again for your constructive suggestions! We hope our responses address your concerns.

[1] LIMA: Less Is More for Alignment. 2023.

Reviewer comment (Experiments):
Is it possible to investigate how the performance of Large Language Models (LLMs) using Mol-Instructions is affected by varying the number of training data instances, checking the redundancy, and possibly employing different sampling methods?

Author response:
Thank you very much for your insightful comments. Below are our responses:

**Impact of Reduced Data on LLM Performance**

The main premise of LIMA[1] is that almost all knowledge is acquired during the pre-training phase, and only a limited amount of instruction tuning data is needed to guide LLMs in generating high-quality outputs. For the domain-specific data encompassed in Mol-instructions, LLMs may not have encountered such data during the pre-training stage. Therefore, using fewer instruction data might not adequately equip the model to grasp the response format of this particular dataset. Our goal in constructing Mol-instructions was to comprehensively cover various biomolecular tasks, thereby infusing the model with relevant knowledge. This ensures that even users focusing on specific tasks can effectively utilize this dataset for subsequent research, benefiting the entire community. 
Indeed, the instruction tuning process may not necessarily require the full set of instruction data. I believe your suggestion provides an excellent starting point for future work, which could focus on discussing the most effective selection methods for sampling the most beneficial alignment data.

Thank you again for your constructive suggestions! We hope our responses address your concerns.

[1] LIMA: Less Is More for Alignment. 2023.

Reviewer comment (Experiments):
The current comparison and presentation scheme in the experiments can lead to overestimated results and misleading conclusions because it ignores some important domain-adapted baselines and mainly compares them with domain-agnostic baselines, which are not fine-tuned or designed for scientific tasks. The authors are encouraged to appropriately acknowledge and compare these tasks with previous methods and datasets in order to improve their potential impact.

Author response:
We sincerely thank you for your insightful feedback. Below are our detailed responses to your concerns: 

**1. More domain-specific baselines**

Based on your recommendations, we have expanded our set of baselines to include Text+Chem T5[1], Galactica[2], and MolT5[3] as these models have undergone instruction tuning, aligning them more closely with the objectives of our study. 

Regarding the baselines [4,5,6], we have chosen not to include them in our comparison as they focus on pretraining purely on molecular data without any instruction-following. This difference in approach makes a direct comparison somewhat unfair. However, we would like to clarify that our evaluation methods for reagent prediction, forward reaction prediction, and retrosynthesis are consistent with those used by Text+Chem T5[1], which we hope alleviates some of your concerns.

For protein-related tasks, the most closely related baseline we found was [7]. However, due to the unavailability of their code, we were unable to include it in our direct comparison. Instead, we have incorporated Galactica[2] as a baseline for these tasks, considering its relevance and accessibility.

The updated results reflecting these expansions can be found in **Tables 3 and 4**, as well as **Figures 5, 6, 8, 9, 10, and 11** in our revised manuscript.

Reviewer comment (Experiments):
Missing comparison to an easier molecule instruction tuning dataset: Text+Chem T5 [1], which has included most of the tasks/datasets in Mol-Instructions, including forward reaction prediction, retrosynthesis, description-guided molecule design, and molecule description generation. The discussion and comparison to Text+Chem T5 are missing.

Author response:
We sincerely thank you for your insightful feedback. Below are our detailed responses to your concerns: 

**1. More domain-specific baselines**

Based on your recommendations, we have expanded our set of baselines to include Text+Chem T5[1], Galactica[2], and MolT5[3] as these models have undergone instruction tuning, aligning them more closely with the objectives of our study. 

Regarding the baselines [4,5,6], we have chosen not to include them in our comparison as they focus on pretraining purely on molecular data without any instruction-following. This difference in approach makes a direct comparison somewhat unfair. However, we would like to clarify that our evaluation methods for reagent prediction, forward reaction prediction, and retrosynthesis are consistent with those used by Text+Chem T5[1], which we hope alleviates some of your concerns.

For protein-related tasks, the most closely related baseline we found was [7]. However, due to the unavailability of their code, we were unable to include it in our direct comparison. Instead, we have incorporated Galactica[2] as a baseline for these tasks, considering its relevance and accessibility.

The updated results reflecting these expansions can be found in **Tables 3 and 4**, as well as **Figures 5, 6, 8, 9, 10, and 11** in our revised manuscript.

Reviewer comment (Experiments):
Missing comparison to scientific language models such as Galactica [2], which is specifically designed and pretrained for the science corpus and should be considered an important baseline compared to the general-purpose LLMs (e.g., Vicuna and Alpaca) reported in the experiments.

Author response:
We sincerely thank you for your insightful feedback. Below are our detailed responses to your concerns: 

**1. More domain-specific baselines**

Based on your recommendations, we have expanded our set of baselines to include Text+Chem T5[1], Galactica[2], and MolT5[3] as these models have undergone instruction tuning, aligning them more closely with the objectives of our study. 

Regarding the baselines [4,5,6], we have chosen not to include them in our comparison as they focus on pretraining purely on molecular data without any instruction-following. This difference in approach makes a direct comparison somewhat unfair. However, we would like to clarify that our evaluation methods for reagent prediction, forward reaction prediction, and retrosynthesis are consistent with those used by Text+Chem T5[1], which we hope alleviates some of your concerns.

For protein-related tasks, the most closely related baseline we found was [7]. However, due to the unavailability of their code, we were unable to include it in our direct comparison. Instead, we have incorporated Galactica[2] as a baseline for these tasks, considering its relevance and accessibility.

The updated results reflecting these expansions can be found in **Tables 3 and 4**, as well as **Figures 5, 6, 8, 9, 10, and 11** in our revised manuscript.

Reviewer comment (Experiments):
Missing comparison to molecular language models such as MoMu [3] and MolT5 [4], which are designed for description-guided molecule design and molecule description generation.

Author response:
We sincerely thank you for your insightful feedback. Below are our detailed responses to your concerns: 

**1. More domain-specific baselines**

Based on your recommendations, we have expanded our set of baselines to include Text+Chem T5[1], Galactica[2], and MolT5[3] as these models have undergone instruction tuning, aligning them more closely with the objectives of our study. 

Regarding the baselines [4,5,6], we have chosen not to include them in our comparison as they focus on pretraining purely on molecular data without any instruction-following. This difference in approach makes a direct comparison somewhat unfair. However, we would like to clarify that our evaluation methods for reagent prediction, forward reaction prediction, and retrosynthesis are consistent with those used by Text+Chem T5[1], which we hope alleviates some of your concerns.

For protein-related tasks, the most closely related baseline we found was [7]. However, due to the unavailability of their code, we were unable to include it in our direct comparison. Instead, we have incorporated Galactica[2] as a baseline for these tasks, considering its relevance and accessibility.

The updated results reflecting these expansions can be found in **Tables 3 and 4**, as well as **Figures 5, 6, 8, 9, 10, and 11** in our revised manuscript.

Reviewer comment (Experiments):
Missing comparison to recent methods for reagent prediction [7], forward reaction prediction [5,6], and retrosynthesis [5], noting that the baselines in these domains employ different evaluation metrics compared to those reported in the paper.

Author response:
We sincerely thank you for your insightful feedback. Below are our detailed responses to your concerns: 

**1. More domain-specific baselines**

Based on your recommendations, we have expanded our set of baselines to include Text+Chem T5[1], Galactica[2], and MolT5[3] as these models have undergone instruction tuning, aligning them more closely with the objectives of our study. 

Regarding the baselines [4,5,6], we have chosen not to include them in our comparison as they focus on pretraining purely on molecular data without any instruction-following. This difference in approach makes a direct comparison somewhat unfair. However, we would like to clarify that our evaluation methods for reagent prediction, forward reaction prediction, and retrosynthesis are consistent with those used by Text+Chem T5[1], which we hope alleviates some of your concerns.

For protein-related tasks, the most closely related baseline we found was [7]. However, due to the unavailability of their code, we were unable to include it in our direct comparison. Instead, we have incorporated Galactica[2] as a baseline for these tasks, considering its relevance and accessibility.

The updated results reflecting these expansions can be found in **Tables 3 and 4**, as well as **Figures 5, 6, 8, 9, 10, and 11** in our revised manuscript.

Reviewer comment (Experiments):
Although incorporating all methods for empirical comparison is challenging and an LLM is not expected to outperform all state-of-the-art methods in every task, comparison to Text+Chem T5 [1] and Galactica [2] is essential and important due to their similar purposes to this submission. It would also be valuable to report performances of single-task state-of-the-art methods in the Tables and Figures (even without outperforming them) to help empirical comparisons in follow-up works.

Author response:
We sincerely thank you for your insightful feedback. Below are our detailed responses to your concerns: 

**1. More domain-specific baselines**

Based on your recommendations, we have expanded our set of baselines to include Text+Chem T5[1], Galactica[2], and MolT5[3] as these models have undergone instruction tuning, aligning them more closely with the objectives of our study. 

Regarding the baselines [4,5,6], we have chosen not to include them in our comparison as they focus on pretraining purely on molecular data without any instruction-following. This difference in approach makes a direct comparison somewhat unfair. However, we would like to clarify that our evaluation methods for reagent prediction, forward reaction prediction, and retrosynthesis are consistent with those used by Text+Chem T5[1], which we hope alleviates some of your concerns.

For protein-related tasks, the most closely related baseline we found was [7]. However, due to the unavailability of their code, we were unable to include it in our direct comparison. Instead, we have incorporated Galactica[2] as a baseline for these tasks, considering its relevance and accessibility.

The updated results reflecting these expansions can be found in **Tables 3 and 4**, as well as **Figures 5, 6, 8, 9, 10, and 11** in our revised manuscript.

Reviewer comment (Experiments):
The quality control and curation process for constructing the dataset should be described more thoroughly, as more details in data processing would strengthen confidence in the dataset.

Author response:
**2. Lack of data processing details**

Due to page constraints, a detailed description of our data transformation process, from the original sources to instruction data, was not feasible within the main text. However, to ensure transparency and reproducibility, we have included an extensive explanation in **Appendix B (Pages 15-22)**. This section thoroughly details each task's definition, data sources, and the methodology employed to convert this data into the intended instruction format. We believe this provides a clear and comprehensive understanding of our data processing approach. Moreover, we have also added further details on quality control measures **(Highlighted in Section 3.2, Page 4)**.

Reviewer comment (Experiments):
The Mol-Instructions dataset integrates various tasks in the field of biochemistry, such as the molecular and protein-related characteristics shown in Table 2; however, the experiments only conduct tests on a limited number of biomolecular tasks. This evaluation approach may not fully reflect performance on complex biomolecular problems.

Author response:
**3. Insufficient tasks**

We apologize for any misunderstanding and would like to clarify that our experiments comprehensively cover the 17 tasks listed in Figure 3. These include:

- **Molecule-oriented Tasks:** Molecular Description Generation (Figure 5), Description-guided Molecule Design (Table 4), Forward Reaction Prediction (Table 4), Retrosynthesis (Table 4), Reagent Prediction (Table 4), Property Prediction (Table 3).
- **Protein-oriented Tasks:** Domain/Motif Prediction (Figure 5), Functional Description Generation (Figure 5), Protein Function Prediction (Figure 5), Catalytic Activity Prediction (Figure 5), Protein Design (Figure 9)
- **Biomolecular Text Tasks:** Open Question (Figure 6), Multi-choice Question (Figure 6), Chemical-protein Interaction Extraction (Figure 6), Chemical-disease Interaction Extraction (Figure 6), Chemical Entity Recognition (Figure 6), True or False Question (Figure 6).

We believe that the variety of tasks covered by Mol-Instructions provides a substantial representation of common biomolecular challenges. Nevertheless, we recognize that there are aspects of biomolecular research that we have not yet explored. These represent opportunities for further enhancement in future research, and we are committed to continually expanding and refining our dataset to encompass an even wider range of biomolecular challenges.