Paper: ProteiNexus: Illuminating Protein Pathways through Structural Pre-training (ICLR-2024)

Reviewer comment (Novelty):
The majority of the components within the proposed method are adaptations from prior studies, which the paper fails to acknowledge. The transformer approach mirrors that of [1], and the pre-training objective resembles [2]. Consequently, the work's novelty is somewhat questionable.

Author response:
- The novelty of our work lies in the protein structure encoder, the design of the pretraining method, and the generality of the pretrained model. Our approach is applicable to a wide range of downstream tasks, such as affinity prediction, antibody sequence design, and generation of corresponding CDR structures. These are areas in protein representation learning that have not been extensively explored before. 
  - In reference [1], the initial features of residues are based on the local geometric structure of the Cα atoms, and the atomic coordinate information is considered only within the residues as a supplementary structural feature. In contrast, we construct the position direction between any residues i and j using a spatial position encoder. 
  - Additionally, the distance encoder provides distance information between any backbone atoms. We no longer treat atomic-level coordinate information as an auxiliary property of local residue structure, but instead calculate it across the entire protein structure. The pairwise relationships constructed in this way complement the sequence-based self-attention mechanism, continuously updating the model's parameters to enhance representational capacity and performance. In addition, our pretraining objective also differs slightly from the distance prediction, angle prediction, and dihedral prediction mentioned in [2]. As far as I know, those approaches are conducted at residue level. In our case, we recover the distances between atoms by using residue-level pairwise representations, ensuring that while capturing interactions between residues, we can preserve atomic-level fine-grained information as much as possible.

Reviewer comment (Novelty):
The paper's "related work" section is not exhaustive. Notable omissions in the protein language domain include TAPE [3], ProtTrans [4], Ankh [5], and ProGen2 [6]. Additionally, recent advancements in protein structure representation, such as ProNet [7] and CDConv [8], are overlooked. A discussion on the connection between this study and previous ones is conspicuously absent.

Author response:
We appreciate your valuable feedback and acknowledge the oversight in our "related work" section. We recognize the significance of including recent advancements in protein language models like TAPE, ProtTrans, Ankh, and ProGen2, as well as advancements in protein structure representation such as ProNet and CDConv. In the revised manuscript, we will ensure to provide a comprehensive discussion on the connection between our study and these relevant works to enhance the overall context of our research. Thank you for bringing these important references to our attention.

Reviewer comment (Experiments):
The authors motivate the use of transformers as encoders by critiquing graph-based representation learning methods for two reasons: 1) their inability to capture detailed atomic information, and (2) their disregard for long-range interactions. However, recent studies [7,9,10] demonstrate that graph-based methods can indeed be extended to the atom level. Additionally, the authors' method focuses solely on backbone-level structures, thus overlooking side-chain details. Regarding long-range interactions, the paper lacks experiments substantiating their claim. Thus, the critiques they raise against graph-based methods lack evidence.

Author response:
- For example, in [7,10], the authors represent protein structures using graphs, where each amino acid is treated as a node and edges are defined based on a cutoff radius. This is a common construction method in graph representation learning. However, it often ignores the interactions between distant residues in inter-chain interactions, as the distance scale differs between intra-chain and inter-chain residues. In contrast, our approach considers both the distance and direction information between any two residues, reducing the impact of the cutoff radius on capturing long-range interactions. 
- As you mentioned, the details of side chains are worth exploring, and we have also taken note of this issue. Building all-atom model of proteins is challenging due to the different types and numbers of atoms in different residue side chains. Moreover, considering the low degrees of freedom in side chain atoms, introducing side chain atoms would increase computational complexity without matching the benefits. In [7], the all-atom model added four torsion angles to the backbone atom model but did not model all side chain atoms explicitly like backbone atoms. Instead, we aggregate the position information of side chain atoms into an additional virtual atom representation. We used the same training settings as the backbone atom pretraining and tested it on downstream tasks. The results showed that incorporating side chain information increased computational complexity without significantly improving the model's predictive performance. We hope to find a more reasonable approach to extend it to all-atom representation in future work.

|         | fold  | super | family | EC   | CATH |
|---------|-------|-------|--------|------|------|
| All-atom| 45.5  | 78.3  | **98.2** | **88.7** | 50.7 |
| ProteiNexus| **47.6** | **79.7** | 98.0 | **88.4** | **53.5** |

Reviewer comment (Evaluation):
The model quality assessment, used by the authors for evaluation, has a potential data leakage risk during pre-training. This task aims to predict the GDT-TS score of certain model predictions without revealing the ground-truth structures. Yet, using the PDB data up to May 1st, 2023, means the model may have encountered target structures from CASP14 and CASP15 during pre-training. Despite different loss functions, this could pose a significant issue.

Author response:
Thank you for highlighting the potential risk of data leakage in the model quality assessment. Due to the distinct objectives and loss functions between pre-training and model quality assessment tasks, we inadvertently overlooked the presence of data overlap between the pre-training dataset and the test dataset. Upon examination, we identified that 6/42 and 3/20 native protein structures in the CASP14 and CASP15 test sets, respectively, had appeared in the pre-training dataset. Due to the extensive time required for a complete retraining, it is not feasible to directly assess the impact of data leakage on the results within the allotted discussion time for this session. 
 - However, the following experiments are designed to alleviate such concerns to some extent. In Table 6 of Appendix C.1, we present the test results on CASP14 and CASP15 for DeepAccNet(40 times the size of our fine-tuning data) trained with and without pre-training. It is evident that, without the influence of pre-training data, our approach has already achieved state-of-the-art results on 6 out of 8 (3 out of 8) evaluation metrics for the CASP14 (CASP15) test set. Nevertheless, we acknowledge the importance of rigorous scrutiny and additional experiments to address concerns related to data leakage in this task. We will supplement these experimental results in the appendix.

Reviewer comment (Experiments):
In the model quality assessment, the authors omit essential baselines. Notably, this task has been included in the Atom3D benchmark [11], where baselines [9,12] are essential references.

Author response:
Thank you for your comprehensive consideration. Indeed, it is important to incorporate more representative baselines in the task of model quality assessment. Given the multitude of metrics available for evaluating protein model quality, we opted for the most universally recognized measures, GDT-TS and LDDT. Ornate[12] assesses model quality by fitting CAD scores. Therefore, including Ornate as one of our baselines might not be entirely fair to its approach. We will continue to research and consider other suitable baselines to ensure a comprehensive evaluation and comparison of different methods in the task of model quality assessment.

Reviewer comment (Experiments):
For binding affinity prediction, the authors neglect to explain their dataset splits—critical to avoid data leakage. Given the small test datasets, it is standard to conduct multiple cross-validations under varying random seeds. Traditional methods like FlexDDG [13] should also be considered for comparison.

Author response:
In the prediction of binding affinity, we consider each mutation to be unique, and thus, we employ a random split of the dataset into training, validation, and testing sets with a ratio of 9:1:1. While a cross-validation approach would be more reasonable, we plan to further optimize the experimental results in subsequent iterations.

Reviewer comment (Experiments):
In the EC and fold classification benchmarks, there is an absence of vital baselines, notably CDConv [8] and ESM-GearNet [14]. The authors might also explore the more challenging GO prediction tasks detailed in [15]. Even without these benchmarks, the authors' method falls behind leading approaches.

Author response:
Thank you for acknowledging the additional outstanding baselines we provided for comparison. While achieving state-of-the-art results on the EC classification task, ProteiNexus is not designed solely for protein structure or function classification, unlike most previous protein representation learning approaches. Our goal is to find a universal paradigm to address a range of protein-related computational problems and extend its applicability to fields like drug discovery that have practical value. Of course, improving the accuracy of classification task predictions and GO predictions is also a goal for our future work. We hope to further demonstrate the versatility of our model.

Reviewer comment (Experiments):
For protein design tasks, there are serious data leakage problems due to the presence of test data in the pre-training dataset. As discussed in App. E.3, such leakage can dramatically affect performance. The authors have not provided a fair comparison with other methods, which makes the evaluation here not convincing.

Author response:
As described in Appendix 3, we believe that the decline in protein design performance is not solely attributed to data leakage; a substantial portion is due to the uneven distribution of pre-training data. We will conduct extensive experiments to validate this hypothesis.

Reviewer comment (Novelty):
Although the paper is well-written, logically coherent, and self-consistent, I'm afraid the novelty of the paper is not too high. The pertaining strategy which combines sequential and structural information is not new to the field. Furthermore, the major contribution of the paper, which is encoding both the atom-level and the finer-grained distance information, has also been studied extensively in recent years. Therefore, I cannot be persuaded of the novelty of the manuscript unless the authors can provide more evidence about how their model differentiates from existing methods and how their adaptations contribute to the enhanced model performance.

Author response:
I consider the novelty of this work lies in the proposed protein structure encoder, pretraining strategy, and the generality of the pretrained model. 

 - While there have been many works in the field of protein structure representation learning, their adaptability to various tasks is not straightforward. Most of these works excel in protein structure or functional classification tasks, without further exploration in a broader range of tasks. However, our approach demonstrates the ability to extend a simple and effective model to a wide range of tasks, including classification, regression, sequence generation, and structure generation, while consistently achieving promising results.
 - Although various protein structure encoders have emerged, adapting to protein complexes remains challenging. By employing a concise yet efficient encoder, we have successfully established the relative positional relationships between residues, aided by distance relations between pairs of backbone atoms. Without the need for intricate modeling, our model vividly presents intra-chain contacts and inter-chain interactions in pair representations. In tasks involving protein-protein interactions, such as binding affinity prediction and antibody design, our model showcases its accurate modeling capabilities for interactions within protein complexes.
 - For the pretraining strategy, in addition to the mixed noise, we introduced a novel self-supervised task. Unlike the simple denoising task, we aimed to recover accurate interatomic distances of residue-level pair representations that incorporate information from a hierarchical structure. In a more challenging pretraining environment, the model is forced to learn more extensive and in-depth feature representations.

Reviewer comment (Experiments):
For pretraining experiments, I'm wondering how the noise level is determined. Besides, I'm wondering whether the authors have evaluated the data efficiency of the proposed pertaining approach by varying the pertaining data sizes.

Author response:
We have detailed our chosen noise levels in Table 11 in Appendix D.2. We use different probabilities to mask the proportions of residues and determine the level of noise introduced. Specifically, we employ probabilities of 0.6, 0.2, and 0.2 to mask 15%, 50%, and 100% of sequence lengths, respectively. Additionally, we apply varying degrees of perturbation to the atom coordinates of these masked residues. More specifically, we introduce noise with probabilities of 0.2 and 0.8, following normal distributions with parameters N(0,0.1) and N(0,1), respectively. This introduces noise in the spatial relationships between atoms.

 - Simultaneously, we conducted an in-depth investigation into the impact of different noise levels on pretraining effectiveness. Firstly, we explored uniformly distributed noise within the range of (-1,1). Secondly, we adjusted the proportion of protein residues being masked and increased the probability of completely masking sequence lengths. We applied masking to 15%, 50%, and 100% of sequence lengths with probabilities of 0.3, 0.2, and 0.5, respectively. The results indicate that the strategy of masking residues and the introduced noise level significantly influence pretraining effectiveness. Increasing the difficulty of the pretraining task contributes to enhancing the model's generalization. 

|            | fold | super | family | EC  | CATH |
|------------|-------|-------|--------|------|------|
| noise-level| 34.1 | 52.7 | 95.4   | 78.0   | 49.6 |
| mask-level | **50.8** | **81.7** | **98.1** | 87.2 | **55.9** |
| ProteiNexus| 47.6 | 79.7 | 98.0 | **88.4** | 53.5 |

 - Regarding the evaluation of the impact of pretraining data on downstream tasks, we have not yet experimented with changing the size of the pretraining data. From the results shown in Experiment 6 of Table 13 and Table 14, it is evident that the size of the pretraining data has a significant influence on the results. We will conduct more detailed evaluations by selecting different sizes of pretraining data and applying different levels of redundancy reduction to assess their effects.

Reviewer comment (Evaluation):
The authors mention in the conclusion that "... an efficient pre-training model ...", but I'm wondering how the "efficiency" is illustrated: does it enhance model performance, have less computation cost, or require less training data?

Author response:
Our efficiency is primarily manifested in several aspects. 
 - Firstly, we are capable of addressing various protein computation tasks based on a universal pre-trained model, eliminating the need for designing specific model architectures for each particular task. 
 - Secondly, there is an enhancement in model performance. Our pre-training approach aims to improve the representation learning process, allowing for the elevation of downstream task performance by effectively utilizing protein structure data, even in scenarios where annotated training data is scarce.