Paper: NL2ProGPT: Taming Large Language Model for Conversational Protein Design (ICLR-2024)

Reviewer comment (Evaluation):
Evaluation: the method seems to use the same model in the framework for evaluation. For instance, they use ESM2 to embed the structure with reward constraints to an ESM-based cluster and use ESMFold (built on ESM2) for structure prediction evaluation. Also, they use the consistency (with Rosetta) reward in Step 3 to constrain the model and evaluation. This may raise the question of model performance benefits from the inductive bias of these pretrained models and tools.

Author response:
We sincerely do not agree that our evaluation is unfair because ESMFold is only used for structure prediction, and the real score is obtained by the Rosetta scoring function. In addition, we do not think that using RL is unfair for comparison. First, we use multiple evaluation metrics to verify our model’s superior performance than other methods as shown in Figure 2. Second, we also report the performance of our method without RL in Table 1, still showing a better performance than the baseline model.

Reviewer comment (Evaluation):
The paper claims to embed the structural information into the description, but it’s doubtful how much the structure is preserved. First, though the ESM2 paper claims their embeddings have structural information, it is still implicit. Second, though the case study shows some insight into the cluster representation, it’s unclear how much information UMAP (into 2-D) and k-mean can preserve, as we know the loss of information after the dimension reduction and the difficulty of clustering.

Author response:
From Table 2, we can see that our method can preserve quite high structural information in the generated proteins. It also should be emphasized that we provide a new **flexible** way to incorporate structural information for the protein language model that has not been explored before.

Reviewer comment (Evaluation):
In Figure 2, it doesn’t seem the proposed method achieves the best performance in any measure. For instance, a similar approach — ESM2-MR model is closer to the GT and performs better in the first one.

Author response:
It is important to clarify that in Figure 2, **'GT' represents the ground truth protein**.

Reviewer comment (Reproducibility):
For step 1, how do you improve the diversity with ChatGPT? Do you also input the protein sequence to ChatGPT?

Author response:
We exclusively leverage ChatGPT to enrich the textual description of proteins. Only text is inputted; protein sequences are not included.

Reviewer comment (Writing):
For step 2, what are the input and output? From the figure, it seems like a pretraining step with self-supervision (next-token prediction). Still, the description in the paper says it’s p(a|w), meaning predicting amino acids from the description. Can you elaborate on this step?

Author response:
Our methodology involves concatenating protein textual descriptions with protein sequences for unified input during self-supervised training. Our focus is solely on obtaining protein sequences through text. During inference, protein sequences are fully derived from the model through textual descriptions.

Reviewer comment (Reproducibility):
For step 2, what is the initial state of GPT2? Which checkpoint is that?

Author response:
We utilize the official GPT-2 checkpoints provided by Hugging Face: https://huggingface.co/gpt2.

Reviewer comment (Reproducibility):
For step 4, how do you control the diversity of generated sequences given an input protein?

Author response:
GPT employs sampling during generation, providing control over diversity by adjusting the sampling strategy.

Reviewer comment (Evaluation):
How well do they cluster in 2-d of UMAP?

Author response:
The combination of Figure 4, Table 2, and Table 3 indirectly validates the effectiveness of our clustering approach.

Reviewer comment (Writing):
Terminology: why do you call it conversational protein design? It may be confusing to the dialog or conversation-based LLM.

Author response:
Since our model is developed as a protein generation model within a conversational system, we refer to it as conversational protein design.

Reviewer comment (Experiments):
I don't quite understand how ChatGPT is used to generate the text descriptions. It says something like: "We then feed these constructed templates into ChatGPT OpenAI (2023) to obtain diverse protein text descriptions by using several prompts. These descriptions constitute the training dataset for text-protein pairs, serving as a foundation for further research and analysis." In Section 4.1 it also says: "Our training dataset comprise 1,001,890 text-protein sequence pairs in total." Are these training examples from the above process with ChatGPT? If so, how did you do any verification on the quality of this dataset?

Author response:
We enriched textual descriptions using ChatGPT, specifically by leveraging the generated text from templates through ChatGPT to diversify the textual descriptions. For instance, the original statement "Provides a protein that contains GNAT domain and belongs to ESM_51 and ONTO_38." was transformed into "This protein presents attributes encompassing the GNAT domain and falls under the classifications of ESM_51 and ONTO_38." This process focuses solely on **enhancing the diversity of textual descriptions**, ensuring that the quality of the dataset remains unaffected.

Reviewer comment (Experiments):
Given that some of the models use text as inputs like the authors' approach and some do not (e.g., Progen), I am a bit confused as to how all the models are compared. Is each model fed a different input and what are these inputs?

Author response:
In fact, we mainly compare the **quality** of generated protein sequences through three metrics, i.e., conformational energy distributions, foldability-measured sequence pLDDT distributions and self-consistency distributions. Therefore, for those methods without text inputs, the protein sequences are generated without text constraints.

Reviewer comment (Evaluation):
When evaluating for generality and consistency, are the metrics used the same as those used for RL? If so, it would be unfair since the model would be overfitting on the reward. Some clarification would be great.

Author response:
We sincerely do not think that using RL is unfair for comparison. First, we do not find any overfitting evidence in our experiment results. Second, we use multiple evaluation metrics to verify our model’s superior performance than other methods as shown in Figure 2. Third, we also report the performance of our method without RL in Table 1, still showing the better performance than the baseline model.