Paper: Illuminating Protein Function Prediction through Inter-Protein Similarity Modeling (ICLR-2024)

Reviewer comment (Theory):
Theoretical Contribution: The manuscript could be improved by providing a deeper theoretical understanding or insights into why certain methods perform better and under what conditions each approach is preferable.

Author response:
We want to argue that this paper is an application-oriented paper, where we put our focus on a practical function annotation setting. Our proposed method is principal as derived in Sec. 3 and achieves good performance on our benchmark. We believe the contribution of this paper is enough for a conference paper and we leave the theoretical analysis as future work.

Reviewer comment (Novelty):
Innovation Clarification: The authors should clearly articulate any novel contributions of their work, particularly if there are innovative aspects beyond the combination of predictive and retriever-based methods.

Author response:
Thanks for the question! We add a paragraph in the Introduction and Related Work sections to discuss the contribution of this work. 

Specifically, we find that there is still a lack of systematic study of modern deep learning methods and retriever-based methods for function annotation. The most closely related work should be CLEAN [1], where they benchmark their proposed supervised contrastive learning methods with several retriever-based methods. However, there are still many deep learning baselines not considered in that paper. In this paper, we include more modern baselines in our benchmark and add another experiment on the CLEAN setup to show the advantage of our method in App. E.

From method perspective, there are many existing works using retriever-based and transductive learning methods for other machine learning applications, but there has been little effort put in the protein function annotation task. One related work along this direction is a concurrent work - RSA [2], which studies other sequence-based property prediction tasks. Our work is among the first to study how to use retriever-based methods for enhancing predictor-based methods.  

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

[2] Ma, Chang, et al. "Retrieved Sequence Augmentation for Protein Representation Learning." bioRxiv (2023): 2023-02.

Reviewer comment (Theory):
Theoretical Insights: A section discussing the theoretical implications of the findings would enrich the paper. This could include a discussion on the conditions under which each method excels or fails.

Author response:
We want to argue that this paper is an application-oriented paper, where we put our focus on a practical function annotation setting. Our proposed method is principal as derived in Sec. 3 and achieves good performance on our benchmark. We believe the contribution of this paper is enough for a conference paper and we leave the theoretical analysis as future work.

Reviewer comment (Novelty):
Compared to structure prediction, what's the most significance of function prediction of proteins?

Author response:
Function prediction is clearly an important problem in the protein domain, as the ultimate goal in this domain is to design proteins with desired functions. This problem has been studied for decades [1,2,3]. Structure prediction is just an intermediate step for us to understand protein functions, as structure directly determines protein functions.

[1] Ashburner et al. "Gene ontology: tool for the unification of biology. The Gene Ontology Consortium". Nature Genetics (2000). 25 (1): 25–9. 

[2] Webb et al. “Enzyme nomenclature 1992: recommendations of the Nomenclature Committee of the International Union of Biochemistry and Molecular Biology on the nomenclature and classification of enzymes.” Academic Press (1992)

[3] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

Reviewer comment (Experiments):
The paper lacks a dataset section. While the paper mentions the use of some datasets, they are never adequately described. For example, how many proteins are included, and where were they obtained from? The description of the tasks is also lacking; they are only briefly covered in section 5.1.

Author response:
Thanks for the questions. During rebuttal, we’ve added more dataset details and evaluation setup in App. C.

Reviewer comment (Experiments):
The evaluation procedure is not adequately described in the paper. There is no explanation of how the training and test sets are obtained, or what kind of evaluation is performed.

Author response:
Thanks for the questions. During rebuttal, we’ve added more dataset details and evaluation setup in App. C.

Reviewer comment (Experiments):
The paper claims that their approach reduces computational requirements while needing less running time, yet this is not supported by experiments or any comparisons to other methods. Additionally, it is difficult to properly judge the reported performance without knowing the size of the dataset used.

Author response:
Thanks for the question. In the revised version, we include the time comparison of different methods in App. E.3.2.

Reviewer comment (Experiments):
The paper uses inter-protein similarity to model and predict function, but it does not compare this approach to important and state-of-the-art methods that model inter-protein similarity. For instance:  
* K. Wu, L. Wang, B. Liu, Y. Liu, Y. Wang and J. Li, "PSPGO: Cross-Species Heterogeneous Network Propagation for Protein Function Prediction," in IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 20, no. 3, pp. 1713-1724, 1 May-June 2023, doi: 10.1109/TCBB.2022.3215257  
* Torres, M., Yang, H., Romero, A.E. et al. Protein function prediction for newly sequenced organisms. Nat Mach Intell 3, 1050–1060 (2021). https://doi.org/10.1038/s42256-021-00419-7  
* Shuwei Yao, Ronghui You, Shaojun Wang, Yi Xiong, Xiaodi Huang, Shanfeng Zhu, NetGO 2.0: improving large-scale protein function prediction with massive sequence, text, domain, family and network information, Nucleic Acids Research, Volume 49, Issue W1, 2 July 2021, Pages W469–W475, https://doi.org/10.1093/nar/gkab398

Author response:
Thanks for bringing our attention to these works. We’ve added a paragraph to discuss the relation between our work and this stream of works in App. A.  

To clarify, instead of focusing on inter-protein similarity, these works use protein-protein interaction networks for function annotation, such as STRING. These networks map both direct physical and indirect functional interactions among proteins. A key limitation of these methods is that they are not able to make predictions for newly sequenced proteins absent in existing PPI networks.  
Moreover, knowing protein-protein interactions is essentially a more difficult challenge, as it requires a more comprehensive understanding of protein properties.  
These problems make this line of work hard to use in real-world settings. Therefore, to ensure a fair comparison, we do not include these baselines in our benchmark.  

Instead, during rebuttal, we add CLEAN paper [1] and other corresponding baselines on function annotation for comparison, which is more related with our work. The results are shown in App. E.1, which clearly demonstrates the effectiveness of our methods.

Reviewer comment (Theory):
The authors assume that there is independence between the function labels. This assumption does not hold for protein function prediction using GO terms. The authors should better justify why they make this design decision.

Author response:
This is a good point! We simply treat the function annotation problem as a multiple binary classification problem following common practice in machine learning. This is a reasonable assumption as most functions can be determined by the protein sequence and structure. **Also, we want to emphasize that the focus of this paper is to study general function annotation tasks without any assumption on 
the function-specific characteristics.** Nevertheless, we admit the effectiveness of modeling the tree structure of GO terms and leave this as future work.

Reviewer comment (Writing):
In section 2, authors claim that "A notable advantage of these neural retrievers over traditional methods is their flexibility in fine-tuning for specific functions, as will discuss in next section". It is not clear where this is discussed or pointed out; clarification is requested.

Author response:
Thanks for the question! One advantage of neural retrievers is that they can be fine-tuned for specific functions using contrastive supervised learning, as done in the CLEAN paper [1]. This is exactly what to do to refine the retriever in the ProtIR framework.  

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

Reviewer comment (Writing):
I found retriever/predictor terminology confusing at first. Please define these terms explicitly.

Author response:
Thanks for pointing this out! We’ve added a definition in Sec. 2.1 and 2.2 in the revised version.

Reviewer comment (Novelty):
The related work is almost comically dense, as is unfortunately required by this conference format. But the upshot is that it's very difficult to get a sense for where you think ProtIR falls in terms of novelty relative to all of these prior methods. The final sentences of the second and third paragraphs are the only hints along these lines. Rather than a laundry list of existing methods, I'd prefer that you cite a couple of reviews and then use the space to explain how your method relates to the state of the art.

Author response:
Thanks for the suggestion! We’ve followed your suggestions to reorganize the related work section and added more related work in App. A. We also discuss the contribution of this work in the end of Introduce and Related Work section. 

We find that there is still a lack of systematic study of modern deep learning methods and retriever-based methods for function annotation. The most closely related work should be CLEAN [1], where they benchmark their proposed supervised contrastive learning methods with several retriever-based methods. However, there are still many deep learning baselines not considered in that paper. In this paper, we include more modern baselines in our benchmark and add another experiment on the CLEAN setup to show the advantage of our method in App. E.

From method perspective, there are many existing works using retriever-based and transductive learning methods for other machine learning applications, but there has been little effort put in the protein function annotation task. One related work along this direction is a concurrent work - RSA [2], which studies other sequence-based property prediction tasks. Our work is among the first to study how to use retriever-based methods for enhancing predictor-based methods.  

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

[2] Ma, Chang, et al. "Retrieved Sequence Augmentation for Protein Representation Learning." bioRxiv (2023): 2023-02.

Reviewer comment (Novelty):
I found it striking that almost all of the related work is from the last few years. In fact, people have been working hard on this problem for decades. See, e.g., Melvin PLOS CB 2011 for a retriever-based method that operates on protein embeddings using sequence and structure. Or see work by JP Vert on kernels for protein sequence similarity. This review might be a good place to get a sense for the long history in this general area: https://academic.oup.com/bib/article/19/2/231/2562645

Author response:
Thanks for the suggestion! We’ve followed your suggestions to reorganize the related work section and added more related work in App. A. We also discuss the contribution of this work in the end of Introduce and Related Work section. 

We find that there is still a lack of systematic study of modern deep learning methods and retriever-based methods for function annotation. The most closely related work should be CLEAN [1], where they benchmark their proposed supervised contrastive learning methods with several retriever-based methods. However, there are still many deep learning baselines not considered in that paper. In this paper, we include more modern baselines in our benchmark and add another experiment on the CLEAN setup to show the advantage of our method in App. E.

From method perspective, there are many existing works using retriever-based and transductive learning methods for other machine learning applications, but there has been little effort put in the protein function annotation task. One related work along this direction is a concurrent work - RSA [2], which studies other sequence-based property prediction tasks. Our work is among the first to study how to use retriever-based methods for enhancing predictor-based methods.  

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

[2] Ma, Chang, et al. "Retrieved Sequence Augmentation for Protein Representation Learning." bioRxiv (2023): 2023-02.

Reviewer comment (Novelty):
What are the two or three most closely related prior methods for solving this problem, and how does your method differ from them?

Author response:
Thanks for the suggestion! We’ve followed your suggestions to reorganize the related work section and added more related work in App. A. We also discuss the contribution of this work in the end of Introduce and Related Work section. 

We find that there is still a lack of systematic study of modern deep learning methods and retriever-based methods for function annotation. The most closely related work should be CLEAN [1], where they benchmark their proposed supervised contrastive learning methods with several retriever-based methods. However, there are still many deep learning baselines not considered in that paper. In this paper, we include more modern baselines in our benchmark and add another experiment on the CLEAN setup to show the advantage of our method in App. E.

From method perspective, there are many existing works using retriever-based and transductive learning methods for other machine learning applications, but there has been little effort put in the protein function annotation task. One related work along this direction is a concurrent work - RSA [2], which studies other sequence-based property prediction tasks. Our work is among the first to study how to use retriever-based methods for enhancing predictor-based methods.  

[1] Yu, Tianhao, et al. "Enzyme function prediction using contrastive learning." Science 379.6639 (2023): 1358-1363.

[2] Ma, Chang, et al. "Retrieved Sequence Augmentation for Protein Representation Learning." bioRxiv (2023): 2023-02.

Reviewer comment (Experiments):
Have you explored using alternatives to BLAST as your retriever?

Author response:
Table A: $F_\max$ for the performance of three sequence retrievers on EC and GO. (#/#/#) shows the results on the split (30%/50%/95%), respectively. 
|#Method|EC|GO-BP|GO-MF|GO-CC|
|:----:|:----:|:----:|:----:|:----:|
| BLAST | 0.740 / 0.806 / 0.872 | 0.344 / 0.373 / 0.448 | 0.505 / 0.557 / 0.640 | 0.275 / 0.284 / 0.347 | 
| BLAST (e-value) | 0.741 / 0.814 / 0.874 | 0.382 / 0.411 / 0.467 | 0.473 / 0.528 / 0.609 | 0.371 / 0.390 / 0.430 |\\
| PSI-BLAST | 0.642 / 0.705 / 0.798 | 0.341 / 0.364 / 0.442 | 0.433 / 0.482 / 0.573 | 0.354 / 0.365 / 0.420 |

Thanks for the suggestion! We’ve follow your suggestions to include both baselines during rebuttal, the results of which are shown in Table A.

**About PSI-BLAST:** As shown in the table, PSI-BLAST does not perform better than BLAST on most tasks. Here is our explanation. PSI-BLAST, or position-specific iterated BLAST, is equivalent to running one round of BLASTp and use the resulting MSA to build a profile and thus position-specific score matrix (PSSM) to imply conservation patterns. The PSSM can guide another round of search, which can be repeated iteratively. However, it requires searching against large sequence database to build an effective PSSM, which performs badly in our settings since our retrieval database is simply the training set of ~20k sequences. The same rationale also applies to HHsearch/HHpred, and we note that these methods are good for searching high-quality MSA but not a good kernel for retrieval w.r.t. a limited database.

Reviewer comment (Evaluation):
Why did you use the alignment score rather than the E-value from BLAST?

Author response:
**About e-value:** We’d like to emphasize that e-value and alignment scores have strong correlation, which may result in the similar set of retrieved proteins. The minor difference here is how to use these scores for weighting different samples. For e-value, we need to take the logarithm and then tune the temperature to rescale the similarity. As shown in the table, there is no obvious conclusion whether e-value or alignment scores are better for function annotation.

Reviewer comment (Evaluation):
In the experimental section, the authors mainly present the experimental results, but the analysis is not sufficient.

Author response:
Thanks for the suggestion! We’ve revised the experiment section to add more analysis about the benchmark results. Please refer to Sec. 5.2.

Reviewer comment (Experiments):
In the method section, the definitions of some symbols in this paper are not clear, and in the experiments section, the description of the datasets is not sufficient.

Author response:
We’ve followed your suggestion to add a section for detailed dataset description in App. C during rebuttal.

Reviewer comment (Evaluation):
The authors emphasize the improvement of computational efficiency, but in the experiment, the authors only provided a rough explanation and did not provide specific improvement results.

Author response:
Sorry for the confusion. We’ve added a section to discuss the training time of different methods in App. E.3.2 to show the efficiency of proposed methods.

Reviewer comment (Writing):
In the first paragraph of the section 2.1, how is the representation of protein x defined? How are the sequence and structural information of proteins integrated?

Author response:
We use $x$ to denote a protein instead of its representations. To obtain its representation, we will feed its sequence and structure to any protein encoder, e.g., ESM-2, GearNet or CDConv. We’ve simplified the notation in the revised version.

Reviewer comment (Experiments):
In the last paragraph of the section 2.3, there was no analysis of the setting of parameter τ. How the different settings of this parameter will affect performance?

Author response:
Thanks for the question! We’ve added a new section discussing the effect of the temperature $\tau$ and the number of neighbors $k$ in App. E.4.

Reviewer comment (Writing):
In the section 3.3 on page 4, what is the meaning of const in the formula? The authors did not provide an explanation.

Author response:
The const in the equation denotes the term irrelevant with the predictor $\psi$, which can be omitted in the E-step. We’ve added an explanation in the revised version.

Reviewer comment (Presentation):
The authors should improve the standardization of the paper. For example: (1) In the section 3.3, the label of the first formula is missing. (2) In Figure 2 of the section 5.3, the labels for the axes are missing.

Author response:
Thanks for pointing out these! We’ve fixed these in the revised version.

Reviewer comment (Experiments):
Other than mentioning in one sentence in section 2.3 that encoders are pre-trained on the fold classification task with 16,712 proteins with 1,195 different folds, data sets are not specified anywhere in the paper. What data set did you have available for EC and GO training and testing?

Author response:
Thanks for the question. In the revised version, we add more description about the datasets and splits in the App. C.

Reviewer comment (Experiments):
ProtIR iterates between the encoder model and prediction model fine-tuning steps. But, without knowing on what data this happens and what is the relationship between training and test data, it is difficult to imagine what is happening in the experiments. For example, are fold classification and protein function data sets coming from different distributions? If so, what is the consequence on the experimental results?

Author response:
Thanks for the suggestion. We did include the source of datasets in the original version and we’ve elucidated more about the dataset in the revised version in App. C. Here the fold classification dataset is collected from SCOPe, which is a different dataset with the Swiss-Prot dataset for function annotation.

Reviewer comment (Reproducibility):
The paper is inconsistent when it comes to explaining hyperparameters for the experiments.

Author response:
We add more explanation about hyperparameter setting in App. E.4. Could you elaborate more about where the inconsistency you find in the hyperparameter setting?

Reviewer comment (Experiments):
Using the structure encoders is a major limitation of this approach because it is only applicable to proteins with known structure. Proteins with known structure are a highly biased sample of the protein space and are typically well studied with a good understanding of their functional properties. Thus, the usefulness of the proposed approach is limited.

Author response:
We want to argue that relying on protein structures should not be a limitation of this work. With the advent of AlphaFold2, it is much easier and efficient to obtain accurate structures for most proteins. Also, there have been large amount of protein structures already released in AlphaFold Database, which can be easily searched for.

To illustrate this, we test our structure encoders on the CLEAN datasets in App. E.1. For most proteins in the training (retrieved) sets (Swiss-Prot), we only have the protein sequences instead of structures. To solve the problem, we simply retrieve the predicted structures from AlphaFold Database according to the UniProt ID. Although there are a small portion of proteins without structures, our structure retriever can still outperform the CLEAN method on NEW-392 dataset. This shows that our method can work well on unseen proteins with AlphaFold2-predicted structures.

Reviewer comment (Experiments):
Missing important baseline and citation. "Enzyme function prediction using contrastive learning" is a recent Science paper about retrieval-based predictors for protein functions.

Author response:
Thanks for the suggestions!  We followed your suggestion to discuss more related work and include the test sets in CLEAN paper for evaluation during rebuttal. The results are shown in App. E.1, where the results of all listed baselines are included. As it is not feasible for us to fine-tune predictor-based methods on the Swiss-Prot training set within such a short time, we only consider several retriever-based methods. It can be observed that even without any supervised training on the swiss-prot training set, our proposed retrievers can outperform CLEAN on both the NEW-392 and Price-149 datasets. Also, to utilize our structure retriever on retrieving Swiss-Prot datasets, we retrieve the predicted structures from AlphaFold Database according to the UniProt ID. Although there are a small portion of proteins without structures, our structure retriever can still outperform the CLEAN method on NEW-392 dataset, which clearly demonstrates the potential of retriever-based methods.

Reviewer comment (Experiments):
Missing the ensemble baseline. If a simple ensemble of predictor and retriever can achieve a good performance, then there is no need to do iterative refinement between predictor and retriever.

Author response:
Thanks for the suggestion! We’ve added the ensemble baseline in App. E.3.1. As shown in Table 6, ProtIR clearly outperforms a simple ensemble baseline on all considered tasks. Only with ProtIR, can these predictors achieve comparable performance with the state-of-the-art protein language model-based methods.

Reviewer comment (Experiments):
Missing evaluations on realistic datasets, which are referred to paper "Enzyme function prediction using contrastive learning".

Author response:
Thanks for the suggestions!  We followed your suggestion to discuss more related work and include the test sets in CLEAN paper for evaluation during rebuttal. The results are shown in App. E.1, where the results of all listed baselines are included. As it is not feasible for us to fine-tune predictor-based methods on the Swiss-Prot training set within such a short time, we only consider several retriever-based methods. It can be observed that even without any supervised training on the swiss-prot training set, our proposed retrievers can outperform CLEAN on both the NEW-392 and Price-149 datasets. Also, to utilize our structure retriever on retrieving Swiss-Prot datasets, we retrieve the predicted structures from AlphaFold Database according to the UniProt ID. Although there are a small portion of proteins without structures, our structure retriever can still outperform the CLEAN method on NEW-392 dataset, which clearly demonstrates the potential of retriever-based methods.

Reviewer comment (Experiments):
The key advantage of retriever-based methods is the capability to annotate unseen proteins or unseen functionality. Investigations are needed to support the effectiveness of the proposed methods.

Author response:
Thanks for the suggestions!  We followed your suggestion to discuss more related work and include the test sets in CLEAN paper for evaluation during rebuttal. The results are shown in App. E.1, where the results of all listed baselines are included. As it is not feasible for us to fine-tune predictor-based methods on the Swiss-Prot training set within such a short time, we only consider several retriever-based methods. It can be observed that even without any supervised training on the swiss-prot training set, our proposed retrievers can outperform CLEAN on both the NEW-392 and Price-149 datasets. Also, to utilize our structure retriever on retrieving Swiss-Prot datasets, we retrieve the predicted structures from AlphaFold Database according to the UniProt ID. Although there are a small portion of proteins without structures, our structure retriever can still outperform the CLEAN method on NEW-392 dataset, which clearly demonstrates the potential of retriever-based methods.

Reviewer comment (Novelty):
Beyond existing protein encoders, there are no specific biological designs for protein function annotation problems. More problem-specific characteristics like the tree structure of go terms should be considered in the modeling.

Author response:
Thanks for the suggestion. We agree that including biological design would be helpful for protein function annotation problems. We want to argue that this will not contradict with the general idea introduced in the paper. Our strategy to train neural structure retrievers and iterative refinement framework are flexible for arbitrary protein encoders, which can include any biological design, such as GearNet and CDConv. 
As we want to focus on more general function annotation problems and make it able to generalize to unseen functions, we do not focus on function-specific characteristics like the tree structure for GO terms. We leave this direction as future work.

Reviewer comment (Experiments):
For the experiments, multiple runs are needed to showcase the prediction stability.

Author response:
The experiments are run on quite large training and test sets, where over 1,000 proteins are used for evaluation. For retriever-based methods, as the retrieved proteins remain the same under different random seeds, there is no need to run multiple times. For predictor-based methods, we run GearNet and CDConv on EC multiple times, where the standard errors for all these methods are below 0.003. This shows the stability of our evaluation.