Paper: Pre-training Sequence, Structure, and Surface Features for Comprehensive Protein Representation Learning (ICLR-2024)

Reviewer comment (Theory):
**Some claims are not right.** In the first several lines of page 6, the paper mentioned "Incorporating these characteristics into the point cloud encoder leads to the formation of embeddings that encompass both the surface’s geometric structure and chemical attributes." On the surface, the author only used atom category and distance, which are chemical features. For geometric features, the author may need to calculate Gaussian curvature, mean curvature, heat kernel signature and something like that, which are geometric features. Besides, I think the calim "pre-training sequence, structure, and surface features" might be not true because there is no sequence pretraining stage involved. Instead, this paper used pretrained ESM-1b, and continued pretraining based on this model, which is continual learning.

Author response:
We thank to the valuable comment. As you pointed out, the phrase "incorporating chemical features" is indeed accurate, and we made the corresponding modifications in the manuscript. Moreover, it is correct that our approach takes the form of continual pre-training (https://arxiv.org/abs/2302.03241), so we have included a comment in the publication to address this.

Reviewer comment (Reproducibility):
**The paper missed some important details.** For example, in SDF, we need the atom type for the current vertex and its neighbors. However, the author didn't mention what atoms they used. The four atoms on backbone? Or the full 44-kind atoms as in AlphaFold2?

Author response:
Thank you for your valuable comment. We have added a sentence addressing the use of full atoms in calculating SDF for the molecular mesh obtained using MSMS.

Reviewer comment (Experiments):
**Function evaluation:** It's just a suggestion. One of the most important motivation of this paper is: surface encodes the function of proteins, so involving surface can get better protein representation. Therefore, it's very natural to apply the learned protein representations to prediction the protein functions. However, the paper didn't do such kind of experiments. I suggest the author add some protein function evaluation, such as protein fitness landscape.

Author response:
We appreciate your constructive feedback. We think predicting EC and GO (molecular function, biological process, cellular component) annotations is a task of related to predicting protein functions, and we already have results for these tasks. The protein fitness landscape prediction is a valuable suggestion. To explore this, we can utilize FLIP(https://github.com/J-SNACKKB/FLIP) or PEER (https://github.com/DeepGraphLearning/PEER_Benchmark) benchmarks; however, the benchmarks primarily rely on sequence information. Since our methodology necessitates the use of protein structures, structure prediction would be necessary in cases where proteins do not have existing structures in the benchmarks. Conducting this during the rebuttal period may be challenging, and the accuracy of the predicted structures could influence the performance of downstream tasks. We appreciate the insightful suggestion, and it seems like a worthwhile endeavor for future exploration.

Reviewer comment (Experiments):
In 4.1.1, N= 16384. What does N refers to? Residue number? R refers to this. Number of vertices on surface? But in 5.1, the author mentioned they sampled 500,000 points for each protein. Besides, Does 500,000 sample points refer to the point cloud after downsampling? According to my opinion, using MSMS, there would be 100-120 vertices around each residue, which means 500,000 points at least represents a protein with length 5,000. That is a very long sequence. I don't believe the average minimum length of the pretraining proteins is 5,000.

Author response:
The variable "N" denotes the number of points in the point cloud of the protein molecular surface, and we randomly sampled 16,384 points to input the point encoder in our experiments. Additionally, 500,000 points were generated for SDF training, serving as the SDF points independent of the protein point cloud input. These points and corresponding SDF values are utilized as the target data for INR training.

Reviewer comment (Presentation):
In Figure1, there is surface encoder and sequence encoder. In title 4.1.1, there is protein encoder. In title 4.1.3, there is Transformer encoder. It's kind of confusing. The author may need to unify the name and give an overall description.

Author response:
Thanks for this helpful comment. We updated the manuscript.

Reviewer comment (Experiments):
Why the author used ESM-1b instead of ESM-2. ESM-2 performs better.

Author response:
At the beginning of our research, we employed ESM-1b, more precisely ESM-GearNet-MC, as our baseline. We kept this setup throughout the all experiments to ensure consistency. (Reference: https://openreview.net/forum?id=AAML7ivghpY)
We acknowledge the advice and also think that utilizing ESM-2 may result in enhanced performance. However, considering the time limitations of rebuttal, we intend to evaluate it in future endeavors. In addition, we have included performance for ESM-2 from the cited literature in our baseline for comparison.

Reviewer comment (Reproducibility):
[Reproducibility] Although the authors provided implementation details, they didn’t have any statements regarding public availability. Since the point of pretrained representations is in using them for various downstream tasks, I think that their significance becomes quite small without making ways to obtain the pretrained representations publicly available.

Author response:
We are planning to make our model publicly available.

Reviewer comment (Experiments):
[Ablations] Would it be possible to develop a surface encoder without implicit neural representations? While the authors discussed the advantage of INR compared to previous methods to represent surfaces, current experiments do not show how important INR is compared to them.

Author response:
We appreciate your constructive review. We utilized INR to represent continuous surfaces, which is the crucial property of protein surfaces. Your suggestion to explore alternative methods for 3D surface representation is a valuable idea and could serve as another promising avenue for future work. For instance, representing the surface as a point cloud could allow for incorporating masked point modeling into pre-training.

Reviewer comment (Experiments):
[Experiments] One of the clear weaknesses of the proposed approach is that it eventually relies on pretraining the structure encoder, GearNet. In other words, it's only applicable for downstream tasks where protein structures are known, which might be the most important in a real world. I'm curious whether authors have thought about downstream tasks where only sequences are available and structures are not. It would be interesting to see how the performance of pre-trained models changes with the usage of predicted protein structures (maybe from AlphaFold) instead of the true ones.

Author response:
We appreciate the constructive feedback. While we believe it's a promising topic, the structure and the surface features in our methodology are determined by the structure. Therefore, if the predicted structure is inaccurate, it could have a substantial impact on the performance and efficacy of our approach. We think that experimenting with datasets consisting of predicted structures in the future could lead to intriguing analyses.

Reviewer comment (Writing):
[Introduction] It would be nice to have more explanation on protein surface characteristics, particularly regarding the relationship between structural features and molecular surfaces. Unfamiliar readers might think that surface information is already well-contained within 3D structures, such that cannot clearly understand the need for incorporating surface information for the pre-training.

Author response:
Thank you for your valuable feedback. We added the following paragraph to the introduction. 
"The protein structure can be divided into the atoms comprising the backbone and the side chains. In this context, the protein surfaces are determined by both backbone and side chain atoms. However, traditional protein structure encoders typically process protein 2D graphs or 3D geometric graphs that only contain alpha carbon or backbone atoms, respectively. Consequently, state-of-the-art representations often lack consideration for side-chain information."

Reviewer comment (Presentation):
[Figure 2] xyz seems to indicate coordinates, but it's inconsistent with notations from the texts.

Author response:
Thanks for your comments. We fixed it and updated the manuscript.

Reviewer comment (Writing):
[Sec 4.1.1.] Does the protein encoder mean point encoder? Due to the prevalent terminologies (point encoder, structure encoder, protein encoder, etc.), there seems to be a little confusing use of terminologies. The authors might want to take a look into the issue.

Author response:
Thanks for your comments. We fixed it and updated the manuscript.

Reviewer comment (Experiments):
[Experiments] Can you further provide the performance of ESM-GearNet-INR? It would help to show the effectiveness of INR with sequence pretraining but without structure pretraining.

Author response:
Thanks for your productive comment. We have conducted the experiments in accordance with the remark, and the results are presented in the table 3 (ESM-GearNet-INR). Interestingly, when comparing with ESM-GearNet-INR-MC, ESM-GearNet-INR exhibited higher performance in several tasks and demonstrated the highest performance on average. When we conducted an experiment where surface pre-training was performed without ESM, we found that the performance improvement was relatively modest compared to structure pre-training alone, so we initially excluded this experiment. However, following your comment, we found that the integration of surface pre-training with ESM resulted in noteworthy enhancements in performance. This result serves as evidence for the importance of the surface modality, and we have added these findings to the manuscript. Thanks for the valuable feedback; it has enriched the content of the paper.

Reviewer comment (Writing):
[Sec 5.3] It would be better to explain the Chamfer distance.

Author response:
Thanks for your productive comment. We added the sentences about chamfer distance in Appendix A.4

Reviewer comment (Evaluation):
INR seems only work well when both structure and sequence information are present. In Table 2, both GearNet-INR-MC and GearNet-INR have similar performance as their counterparts. Only ESM-GearNet-INR-MC shows relatively bigger improvements comparing to ESM-GearNet-MC.

Author response:
We thank the thoughtful comment. Although the addition of surface pre-training without ESM doesn't result in a significant improvement , we have observed an increase in the average performance (GearNet-INR > GearNet, GearNet-INR-MC > GearNet-MC), confirming the importance of incorporating the surface modality in pre-training for protein representation learning. However, despite this, the additional experiment (ESM-GearNet-INR outperforming ESM-GearNet-INR-MC) and the relatively modest performance gains when comparing GearNet-MC and GearNet-INR-MC suggest that further investigation is needed to find the better combination or strategy for integrating all three modalities, particularly the effective integration of surface and structure pre-training. We have clearly stated the need to explore approaches to a better way of combining surface and structure pre-training in the manuscript.

Reviewer comment (Evaluation):
Why does pre-training order impact the EC tasks? Does this observation apply to other downstream tasks as well?

Author response:
Table 3 correction: The experiment in Table 3 was designed to examine the influence of the order of structure and surface pre-training, thus we did not use sequence pre-training. We correct “Pre-training order” part by removing the “sequence” part. Also, the units (such as 0.896 and 0.889) have been revised to 89.6 and 88.9, respectively as in Table 2. The title of the Ablation study has been revised to  "Experiment on Pre-training Order." Additionally, we have conducted and added experimental results for other downstream tasks.

Considering the original results (Table 2) and our additional experiment (ESM-GearNet-INR), it is evident that ESM has the most dominant impact on the downstream tasks. So, when the sequence pre-training information is present, structure and surface play a supportive role in enhancing performance. Table 3 enables us to compare the significance of structure and surface pre-training while excluding the dominant influence of sequence. Based on the results of GearNet-INR (466.1) and GearNet-MC (498.3) shown in Table 2, it can be concluded that in the absence of pre-training on sequences, structure pre-training has a greater influence on downstream tasks compared to surfaces. We conjecture that this observation supports the findings that the structure encoder (GearNet-INR-MC), which learned structure information last, had superior performance compared to GearNet-MC-INR, which was pre-trained in the opposite manner, as shown in Table 3.

Reviewer comment (Experiments):
How does batchsize in finetuning affect the overall downstream performance?

Author response:
We appreciate your constructive feedback. At the early stage of our research, we experimented with different batch sizes (e.g., 8, 16) for several tasks and found no notable differences in performance. So, we consistently conducted the experiments with a batch size of 16 for efficiency and fairness.

Reviewer comment (Experiments):
In section 4.1.4, how is the clamp value decided?

Author response:
We used the clamp value of 0.2, as employed in the DeepSDF paper (https://arxiv.org/abs/1901.05103). We added this sentence in the manuscript.

Reviewer comment (Experiments):
Some more abalations/baselines would clarify the contribution of the surface based features (see questions section).

Author response:
Thank you for your comment. We additionally evaluated the performance of ESM-GearNet-INR, which did not undergo structure pre-training, on downstream tasks and observed the highest level of performance and added this result to Table 2. This serves as additional experimental evidence demonstrating the significance of surface features.

Reviewer comment (Evaluation):
Why do you think is the performance so much dependent on the pretraining order (Table 3). Please clarify this in the paper.

Author response:
Table 3 correction: The experiment in Table 3 was designed to examine the influence of the order of structure and surface pre-training, thus we did not use sequence pre-training. We correct “Pre-training order” part by removing the “sequence” part. Also, the units (such as 0.896 and 0.889) have been revised to 89.6 and 88.9, respectively as in Table 2. The title of the Ablation study has been revised to  "Experiment on Pre-training Order." Additionally, we have conducted and added experimental results for other downstream tasks.  
 
Considering the original results (Table 2) and our additional experiment (ESM-GearNet-INR), it is evident that ESM has the most dominant impact on the downstream tasks. So, when the sequence pre-training information is present, structure and surface play a supportive role in enhancing performance. Table 3 enables us to compare the significance of structure and surface pre-training while excluding the dominant influence of sequence. Based on the results of GearNet-INR (466.1) and GearNet-MC (498.3) shown in Table 2, it can be concluded that in the absence of pre-training on sequences, structure pre-training has a greater influence on downstream tasks compared to surfaces. We conjecture that this observation supports the findings that the structure encoder (GearNet-INR-MC), which learned structure information last, had superior performance compared to GearNet-MC-INR, which was pre-trained in the opposite manner, as shown in Table 3.

Reviewer comment (Experiments):
Does seq->surface->3d structure outperform seq->structure->surface for all tasks? Table 3 only shows EC.

Author response:
Thanks for the helpful comment. We conducted additional experiments on other tasks and found the similar trends with EC task. The results were added in Table 3.

Reviewer comment (Experiments):
In Table 1, it would be nice to have the performance on just ESM.

Author response:
Thank you for your comment. We added ESM-2 (pre-trained on only sequences) results, which are extracted from the reference paper (https://arxiv.org/abs/2301.12040), as a baseline in Table 2.