Paper: MAPE-PPI: Towards Effective and Efficient Protein-Protein Interaction Prediction via Microenvironment-Aware Protein Embedding (ICLR-2024)

Reviewer comment (Novelty):
This paper proposes masking codebook modeling as a pre-training task. Has this approach been used before in other domains (e.g., CV and NLP) or is it the originality of this paper? The authors are encouraged to explain more about the motivation for doing so and provide more experiments to demonstrate its advantages over previous existing masked modeling approaches.

Author response:
We'd like to highlight that Masked Codebook Modeling (MCM) is originally proposed in this paper and is one of our core contributions. We answer your question in three aspects:

- **Novelty.** We appreciate a large amount of works on VAE and masked modeling for CV and NLP. However, MCM is not a simple extension of previous masked modeling methods to VAE-based codebook updating. One of the biggest differences between MCM and related work (including [1] as pointed out by Reviewer wZyc) has been explained in **Lines 139-141**, that
    
    > "Following this line in this paper, but unlike previous works that mask input features or hidden codes [1] for *training a better image encoder or decoder*, we randomly mask the microenvironment codes in the codebook to capture their dependencies for *learning a better codebook*."
    > 

- **Motivation.** The motivation behind MCM is that there are generally overlaps (dependencies) in the microenvironments of two residues that are adjacent in sequence or structure. To capture such microenvironmental dependencies in the codebook, we propose Masked Codebook Modeling (MCM) to mask the codebook $\mathcal{A}$ and then reconstruct the inputs based on the unmasked codes, aiming to learn a more expressive codebook $\mathcal{A}$ rather than a more powerful protein encoder. The above explanations have been described in **Lines 218-222**.

- **Comparison.** We provide a comparison of MCM with two schemes, (1) Masked Input Modeling(MIM) that masks input features {$\mathbf{x}_m$}$\_{m=1}^M$ and (2) Masked Hidden Modeling (MHM) that masks hidden codes {$z_m$}$\_{m=1}^M$, in **Lines 345-351** and **Table 3** of the original manuscript. The reported results have demonstrated the superiority of MCM over MIM and MHM.

[1] Peng Z, Dong L, Bao H, et al. Beit v2: Masked image modeling with vector-quantized visual tokenizers[J]. arXiv preprint arXiv:2208.06366, 2022.

Reviewer comment (Experiments):
Why does MAPE-PPI freeze the protein encoder instead of fine-tuning it?

Author response:
We have explained why MAPE-PPI freeze rather than fine-tune the protein encoder in **Lines 283-284** of the main paper and **Lines 562-569** of **Appendix C**. One main concern is the unaffordable computational overhead of fine-tuning the pre-trained encoder on large-scale PPI datasets. For example, even on the SHS27k dataset containing only 16,912 PPI data, ESM-1b takes more than 10 hours to fine-tune (vs. only ~300s for the FREEZE setting).

Reviewer comment (Experiments):
This paper introduces validation sets for hyperparameter tuning and model selection in the data partition, which makes more sense than the previous data partition. The authors have also reproduced the results of HIGH-PPI using the same data partitioning as MAPE-PPI in Tables 1 and 2, right?

Author response:
You are right. We split the PPIs into the training (60%), validation (20%), and testing (20%) for all baselines (including both MAPE-PPI and HIGH-PPI), as we explained in **Lines 263-266**.

Reviewer comment (Writing):
Some typos need to be corrected.

Author response:
All typos have been corrected in the revised manuscript.

Reviewer comment (Experiments):
Lack of description of the details of the datasets.

Author response:
Due to space limitations, we place a detailed description of the datasets in **Lines 523-534** of **Appendix A**. For more details, please refer directly to the STRING database [1].

[1] Szklarczyk D, Gable A L, Lyon D, et al. STRING v11: protein–protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets[J]. Nucleic acids research, 2019, 47(D1): D607-D613.

Reviewer comment (Experiments):
They just perform parameter sensitivity analysis on one dataset. It remains unclear whether the proposed method is sensitive to the hyper-parameters and how to setup the values in general cases.

Author response:
We have further evaluated the sensitivity of the codebook size $|\mathcal{A}|$ and mask ratio $\frac{|\mathcal{M}|}{|\mathcal{A}|}$ on two additional datasets, SHS148k and STRING, respectively. The detailed experimental results and analysis can be found in **Appendix G**, **Lines 589-593**. Besides, we have described in detail how we determined the hyperparameters in **Lines 266-267** of the main paper and **Lines 539-543** of **Appendix B**. Specifically, the hyperparameters are determined by an AutoML toolkit NNI with pre-defined hyperparameter search spaces. For a fair comparison, the model with the highest validation accuracy will be selected for testing.

Reviewer comment (Experiments):
Lack of description of the details of the datasets.

Author response:
Due to space limitations, we place a detailed description of the datasets in **Lines 523-534** of **Appendix A**. For more details, please refer directly to the STRING database [1].

[1] Szklarczyk D, Gable A L, Lyon D, et al. STRING v11: protein–protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets[J]. Nucleic acids research, 2019, 47(D1): D607-D613.

Reviewer comment (Experiments):
Lack of real world examples to demonstrate the effectiveness of their model.

Author response:
We do not get precisely what is meant by "real-world examples". If it refers to experimental designs, we have tried to make them consistent with real-world scenarios, including real-world Homo sapiens datasets and various data splits to evaluate real-world generalizations and robustness. If it refers to real-world experimental assays in a wet lab, we regret not including them in this paper, as they are quite expensive and time-consuming. As a conference on "representation learning", we are more concerned with the design of computational methods.

If "real-world examples" are very important and our understanding is not right, please do not hesitate to contact us.

Reviewer comment (Theory):
The authors mention that "the complexity of protein structure modeling hinders its application to large-scale PPI prediction", and emphasize it as an efficiency limitation. Why? It is straightforward to see that complexity modeling hinders accurate modeling. However, it is not clear how it hinders the scalability of modeling and why it is inefficient.

Author response:
The ambiguity of the term "complexity" may have led to some misunderstandings. The term "complexity" refers not to the complexity of the protein structure itself in this paper, but to the complexity of the algorithms that model (encode) the protein structure. The complexity of the protein structure does affect the modeling accuracy, but the algorithmic complexity may have a significant impact on the training efficiency. To clarify this point, we have made the following corrections and additions:
- We have replaced “the complexity of protein structure modeling hinders …” with a more concise expression “the training inefficiency of existing structure-based methods hinders …”.
- We provide a time-complexity comparison of the structure-based method (HIGH-PPI) with MAPE-PPI in **Appendix E**, **Lines 576-586**, which explains why HIGH-PPI is inefficient.

Reviewer comment (Novelty):
Important references are missing. (i) https://arxiv.org/pdf/2208.06366.pdf, which does almost the same codebook learning and masked codebook modeling in vision; (ii) https://www.nature.com/articles/s41586-022-04599-z, which define microenvironments of an amino acid residue and do masked "token" prediction. Based on this omitted reference, the innovation of this work is limited. Also, it is inappropriate to claim "propose... MCM", since both microenvironments and MCM are pre-defined in the literature.

Author response:
The two papers [1,2] are relevant to our work, but not as important as you said. We sincerely hope that you do not conclude that our work is "limited innovation" just because these two papers share some similar concepts and terminology with us. We hope that the following comparisons will help you better appreciate our contributions and how we differ from [1,2]:
- Comparison with [1]. We clarify our innovations compared to [1] in three aspects:
   - **Novelty**. It needs to be recognized that hidden codes and codebook are inherently two concepts, the former referring to {$z_m$}$\_{m=1}^M$ and the latter to $\mathcal{A}=${$\mathbf{e}\_i$}$\_{i=1}^{|\mathcal{A}|}$ in this paper. Two kinds of masked modeling are adopted in [1]: (1) Masked Input Modeling(MIM) that masks input features {$\mathbf{x}_m$}$\_{m=1}^M$ and (2) Masked Hidden Modeling (MHM) that masks hidden codes {$z_m$}$\_{m=1}^M$. However, unlike previous works [1] that mask input features {$\mathbf{x}_m$}$\_{m=1}^M$  or hidden codes {$z_m$}$\_{m=1}^M$ for training a better encoder or decoder, we randomly mask the microenvironment codes {$\mathbf{e}_i$}$\_{i=1}^{|\mathcal{A}|}$ in the codebook $\mathcal{A}$ to capture their dependencies for learning a better codebook. The above explanations have been described in **Lines 139-141**.
   - **Motivation.** There are generally overlaps (dependencies) in the microenvironments of two residues that are adjacent in sequence or structure. To capture such microenvironmental dependencies in the codebook, we propose MCM to mask the codebook $\mathcal{A}$ and then reconstruct the inputs based on the unmasked codes, aiming to learn a more expressive codebook $\mathcal{A}$ rather than a more powerful protein encoder. The above explanations have been described in **Lines 218-222**.
   - **Comparison.** We provide a comparison of MCM with two schemes (MIM and MHM) in **Lines 345-351** and **Table 3**, which demonstrates the superiority of MCM over MIM and MHM.

- Comparison with [2]. [2] defines the sequence and structural context around a target residue as its microenvironment and classifies it into **pre-defined** categories based on its physicochemical properties (see Fig. 1(b) in [2]). The essential difference between our work and [2] is that [2] is a microenvironment classification task for known categories, whereas MAPE-PPI aims to discover and encode microenvironments with **unknown** categories. We have clearly stated the differences between our work and [2] in **Lines 58-68**, as follows
    
   > “There has been some work that defines the sequence and structural contexts surrounding a target residue as its microenvironment and classifies it into a dozen types based on its physicochemical properties or geometric features [2], which constitutes a special microenvironment vocabulary. However, unlike the rich word vocabulary in languages, the experimental microenvironment vocabulary is usually coarse-grained and limited in size. Therefore, to learn a fine-grained vocabulary with rich information, we propose …”
   >

We have cited, discussed, and compared [1,2] in the revised manuscript. We sincerely hope that this helps to clarify your concerns on "limited innovation".

---

[1] Peng Z, Dong L, et al. Beit v2: Masked image modeling with vector-quantized visual tokenizers. 2022.

[2] Lu H, Diaz D J, et al. Machine learning-aided engineering of hydrolases for PET depolymerization. 2022.

Reviewer comment (Theory):
The efficiency claim is very confusing. "Extensive experiments show that MAPE-PPI can scale to PPI prediction with millions of PPIs" seems to suggest an inference efficiency. "predict unknown PPIs more efficiently and effectively" also seems to suggest an inference efficiency. However, in the introduction, the authors claim the training efficiency. Also, why the proposed design can enable efficiency? What are the computational and memory overheads of the sufficiently large vocabulary and two-stage training? More detailed time complexity analyses are needed.

Author response:
The efficiency (or computational efficiency) discussed in this paper refers to the training efficiency rather than inference efficiency. As described in subsection 3.1 of the problem statement on **Lines 149-150**, this paper adopts a **transductive** setting (common for graphs), which requires that all test data (without labels) are involved in training. As a result, the main bottleneck here is training efficiency rather than inference efficiency, so in order to "scale to PPI prediction with millions of PPIs", it prefers a model with very high training efficiency. Next, we explain why MAPE-PPI improves efficiency from three aspects：

- **Roles of the Codebook.** The microenvironments of different residues may also be similar or overlap. If the microenvironments of two residues are highly identical, it would be redundant to encode them separately at each training epoch. If we can learn a ``vocabulary" (i.e., codebook) that records those most common microenvironmental patterns, we can use it as an off-the-shelf tool to encode proteins into microenvironment-aware embeddings. Thus, training efficiency can be improved by decoupling end-to-end PPI prediction into two stages of learning the microenvironment codebook and codebook-based protein encoding. These explanations have been added in **Lines 52-58** of the revised manuscript.

- **Training Time Complexity.** A comparison of the time complexity of HIGH-PPI with that of MAPE-PPI is provided in **Appendix E, Lines 576-586**. The key analysis is as follows
> "The time complexity of MAPE-PPI is $\mathcal{O}\big(N\left(|\mathcal{V}_P|\cdot|\mathcal{A}|+|\mathcal{E}_P|\right)+|\mathcal{E}_T|\big)$. Besides, the time complexity of HIGH-PPI for protein encoding and PPI prediction is $\mathcal{O}\big(N\left(|\mathcal{V}_P|+|\mathcal{E}_P|\right)\big)$ (without vector quantization and protein decoding) and $\mathcal{O}(|\mathcal{E}_T|)$, respectively. However, since HIGH-PPI jointly performs protein encoding and PPI prediction in an end-to-end manner (while MAPE-PPI decouples the two), its time complexity is the product of the two, that is $\mathcal{O}\big(N\left(|\mathcal{V}_P|+|\mathcal{E}_P|\right)\cdot|\mathcal{E}_T|\big)$, which is much higher than MAPE-PPI."
>

- **Training Time Analysis.** Refer to **Figure 1** or **Figure A1**, MAPE-PPI trains most efficiently regardless of whether or not pre-training is done and what type of pre-training is used.

Reviewer comment (Experiments):
Only a single level of sequence similarity is considered. More like 30%, 50%, 90% are encouraged. Meanwhile, as the main results in Table 1, multiple runs of experiments are encouraged to show the prediction stability.

Author response:
We have added experiments on how sequence similarity influences performance. The detailed experimental setup, results, and analysis can be found in **Appendix H**, **Lines 594-604**. The issue of prediction stability has been taken into account in the original manuscript, as stated in **Lines 267-268**, that

> "we run each set of experiments five times with different random seeds, and report the average performance.”
>