Paper: Learning to design protein-protein interactions with enhanced generalization (ICLR-2024)

Reviewer comment (Experiments):
I value this study's effort in formulating a more comprehensive evaluation paradigm for mutation effect prediction, particularly in dataset splitting. However, the new splitting strategy is somehow unclear. For instance, the test set has only 5 samples, which are too small. And 3 cross-validation does not align with 5 test folds.

Author response:
We apologize for the confusion. Please see our response to question 1 for detailed clarification. The 5 test samples from SKEMPI v2.0 are difficult to expand due to the high redundancy of the dataset, and these samples are independent from the 3-fold cross-validation. We firstly set aside these 5 representative and independent PPIs for testing, and then split the remaining data into 3 folds.

Reviewer comment (Experiments):
I am a little confused about the evaluation of test folds. As mentioned in Appendix B.2, the author splits the dataset into 3 cross-validation folds based on the "hold-out-proteins feature" and then sets aside 5 random outlier PPIs to form 5 distinct test folds. Firstly, what is the "hold-out-proteins feature"? Please explain that term. Secondly, why are there 3 cross-validation folds but 5 test folds? Commonly, \(k\) cross-validation would result in \(k\) training sets as well as \(k\) validation sets, and usually do not have a so-called test fold. In my understanding, these 5 extra outliers are excluded from both the training and validation sets and are used to evaluate models alone. So here comes the question: how does the author compute the \(\Delta\Delta G\) for these 5 outliers? Notably, 3 cross-validation will lead to 3 sets of models that achieve the lowest losses in 3 different validation sets. Does the author just take an average of these 3 models' output to obtain the binding affinity change? If not, please specify the computational process.

Author response:
We appreciate the detailed analysis of our data preparation and apologize for the confusion. We have now significantly expanded Appendix B.2 “Test Datasets” in the updated paper to address all the raised questions. The intuition of the reviewer about the parts that were unclear is correct, and we briefly answer the questions below.

The “Hold out proteins” is [the column in the SKEMPI v2.0 dataset](https://life.bsc.es/pid/skempi2/database/browse), which specifies similar PPIs. We obtain our three folds by splitting the dataset by the values from this column. This ensures the PPIs are not leaked across folds. After 3-fold cross-validation, we, indeed, average three models for test predictions.

Reviewer comment (Experiments):
How does the author split the Skempi2 by 3 cross-validation? I want to know more details. Also, why do these 5 outlier PPIs are selected randomly? I am afraid that a test set of merely 5 samples is so small. Is there any possibility to increase the test size?

Author response:
Prior to 3-fold splitting, we set aside 5 outlier PPIs for an independent test set. Please note that these outliers are not selected randomly. Instead, they are chosen from a limited pool of PPIs that do not have similar ones in the training data and that have both positive and negative ddG annotations.

To elaborate, we manually examine the PPIs using the information in Figure 2 in the original SKEMPI v2.0 publication [6]. In this figure, connected components composed of two nodes (proteins) represent outlier PPIs. These PPIs do not share a similar binding site or partner with any other PPIs. Among these PPIs, there is a limited number that have both positive and negative ddG annotations. We were able to select 5 such PPIs only after inspecting the vast majority of outliers. This highlights the difficulty of choosing a larger, truly independent test set from SKEMPI v2.0 with a reasonable ddG distribution. We confirmed that the selected test PPIs do not have structural homologs among the other (training) PPIs by calculating interface similarities with iAlign. The maximum iAlign IS-score between all PPIs in the test folds and all PPIs in train-validation folds is 0.22.

Reviewer comment (Evaluation):
Can you report the performance of PPIFormer in the same way as RDE-Net? Namely, the Spearman and Pearson of PPIformer and different baselines in single-mutation and multiple-mutation cases. I hope to see that PPIFormer remains excellent in the multiple-mutation circumstance, which is more challenging and more likely to result in successful protein design.

Author response:
Overall performance:

|          |                | Per-Structure |           | Overall |           |        |        |        |
|----------|----------------|---------|----------|---------|----------|--------|--------|--------|
| **Category** | **Method**         | **Pearson** | **Spearman** | **Pearson** | **Spearman** | **RMSE**   | **MAE**    | **AUROC**  |
| Sequence Based | ESM-1v        | 0.0073  | -0.0118  | 0.1921  | 0.1572   | 1.9609 | 1.3683 | 0.5414 |
|              | PSSM           | 0.0826  | 0.0822   | 0.0159  | 0.0666   | 1.9978 | 1.3895 | 0.5260 |
|              | MSA Transf.    | 0.1031  | 0.0868   | 0.1173  | 0.1313   | 1.9835 | 1.3816 | 0.5768 |
|              | Tranception    | 0.1348  | 0.1236   | 0.1141  | 0.1402   | 2.0382 | 1.3883 | 0.5885 |
| Energy Function | Rosetta       | 0.3284  | 0.2988   | 0.3113  | 0.3468   | 1.6173 | 1.1311 | 0.6562 |
|                | FoldX         | 0.3789  | 0.3693   | 0.3120  | 0.4071   | 1.9080 | 1.3089 | 0.6582 |
| Supervised   | DDGPred        | 0.3750  | 0.3407   | **0.6580** | 0.4687   | **1.4998** | **1.0821** | 0.6992 |
|              | End-to-End     | 0.3873  | 0.3587   | 0.6373  | 0.4882   | 1.6198 | 1.1761 | 0.7172 |
| Unsup./Semisup. | B-factor    | 0.2042  | 0.1686   | 0.2390  | 0.2625   | 2.0411 | 1.4402 | 0.6044 |
|                | ESM-IF        | 0.2241  | 0.2019   | 0.3194  | 0.2806   | 1.8860 | 1.2857 | 0.5899 |
|                | MIF-Dlogit    | 0.1585  | 0.1166   | 0.2918  | 0.2192   | 1.9092 | 1.3301 | 0.5749 |
|                | MIF-Net.      | 0.3965  | 0.3509   | 0.6523  | 0.5134   | 1.5932 | 1.1469 | 0.7329 |
| RDE-PPI      | RDE-Linear     | 0.2903  | 0.2632   | 0.4185  | 0.3514   | 1.7832 | 1.2159 | 0.6059 |
|              | RDE-Net.       | **0.4448** | **0.4010** | 0.6447  | **0.5584** | $\underline{1.5799}$ | $\underline{1.1123}$ | **0.7454** |
| Ours         | PPIformer      | $\underline{0.4281}$ | $\underline{0.3995}$ | $\underline{0.6450}$ | $\underline{0.5304}$ | 1.6420 | 1.1186 | $\underline{0.7380}$ |

Performance on single-point mutations:

|          |                | Per-Structure |           | Overall |           |        |        |        |
|-----------------|------------------|---------|----------|---------|----------|--------|--------|--------|
| **Category** | **Method**         | **Pearson** | **Spearman** | **Pearson** | **Spearman** | **RMSE**   | **MAE**    | **AUROC**  |
| Sequence Based  | ESM-1v           | 0.0422  | 0.0273   | 0.1914  | 0.1572   | 1.7226 | 1.1917 | 0.5492 |
|                 | PSSM             | 0.1215  | 0.1229   | 0.1224  | 0.0997   | 1.7420 | 1.2055 | 0.5659 |
|                 | MSA Transf.      | 0.1415  | 0.1293   | 0.1755  | 0.1749   | 1.7294 | 1.1942 | 0.5917 |
|                 | Tranception      | 0.1912  | 0.1816   | 0.1871  | 0.1987   | 1.7455 | 1.1708 | 0.6089 |
| Energy Function | Rosetta          | 0.3284  | 0.2988   | 0.3113  | 0.3468   | 1.6173 | 1.1311 | 0.6562 |
|                 | FoldX            | 0.3908  | 0.3640   | 0.3560  | 0.3511   | 1.5576 | 1.0713 | 0.6478 |
| Supervised      | DDGPred          | 0.3711  | 0.3427   | 0.6515  | 0.4390   | 1.3285 | 0.9618 | 0.6858 |
|                 | End-to-End       | 0.3818  | 0.3426   | $\underline{0.6605}$  | 0.4594   | $\underline{1.3148}$ | 0.9569 | 0.7019 |
| Unsup./Semisup. | B-factor         | 0.1884  | 0.1661   | 0.1748  | 0.2054   | 1.7242 | 1.1889 | 0.6100 |
|                 | ESM-IF           | 0.1616  | 0.1231   | 0.2548  | 0.1927   | 1.6928 | 1.1671 | 0.5630 |
|                 | MIF-Dlogit       | 0.1053  | 0.0783   | 0.3358  | 0.2886   | 2.5361 | 1.8967 | 0.6066 |
|                 | MIF-Net.         | 0.3952  | 0.3479   | **0.6667** | $\underline{0.4802}$   | **1.3052** | $\underline{0.9411}$ | 0.7175 |
| RDE-PPI         | RDE-Linear       | 0.3192  | 0.2837   | 0.3796  | 0.3394   | 1.5997 | 1.0805 | 0.6027 |
|                 | RDE-Net.         | **0.4687** | **0.4333** | 0.6421  | **0.5271** | 1.3333 | **0.9392** | **0.7367** |
| Ours            | PPIformer        | $\underline{0.4192}$  | $\underline{0.3796}$ | 0.6287  | 0.4772   | 1.4232 | 0.9562 | $\underline{0.7213}$ |

Reviewer comment (Experiments):
In the ablation study, the author claims the benefit of 80% 10% 10% masking proposed by BERT. However, I remember that BERT masked 15% of input tokens and tried to recover them. What does 80%, 10%, 10% mean?

Author response:
BERT indeed masks 15% of tokens. However, for each of the masked tokens, it applies a random transformation as follows: with 80% probability, it leaves the masking token unchanged; with 10% probability, it replaces it with a random token; and with another 10% probability, it leaves the original token. This allows learning more powerful representations of non-masked tokens as it is more robust to noise [3].

Following this idea, we adopt a similar approach. For 80% of the masked residues, we leave the F_0 features as zeros, and for the remaining 20%, we replace them with either the original amino acids or random amino acids. We hypothesize that in the case of proteins, this approach may not only lead to better embeddings of non-masked residues but also serve as a regularization simulating noisy random mutations.

Reviewer comment (Presentation):
In Figure (4), what is the difference between a3 and PPIFormer? It seems a3 is better than PPIFormer, but it is said PPIFormer is the best model. Please specify the difference between the upper and bottom subfigures.

Author response:
We apologize for the confusion. The ablations in the top figure (data ablations, including a3) were run prior to the ones in the bottom figure (model ablations, including PPIformer), using a slightly different subset of SKEMPI v2.0 for zero-shot evaluation. The data ablations were performed on training ddG examples corresponding to two out of three folds from the RDE-Net split [4], while the model ablations were run on the training set from our split. We updated the figure caption for clarification in the updated version of the paper.

Reviewer comment (Theory):
During the phase of mutant effect prediction, the author adopts a simple summation of all log odds ratios. In other words, each substitution of residue is considered independently. I suppose it may be a better solution to calculate the joint probability as \(\log p (\hat{M} = M | \mathbf{c}_{\backslash M})\). Does the author agree with my point?

Author response:
Yes, we agree that predicting the joint probability is highly beneficial over predicting the probability of each single-point substitution independently. The former can enable capturing epistasis (i.e. non-additiveness of mutation effects), which is an abundant phenomenon in proteins [1]. However, please note that in our model, each substitution is not considered independently during prediction. Instead, as shown in Figure 2, the probabilities are estimated in a single forward pass, and only then summed up for mutated positions. While this does not yield a probability from the exact joint conditional distribution $\log{p(\hat{{\bf c_{\it{M}}}} = {\bf c_{\it{M}}} \vert {\bf c_{\setminus \it{M}}})}$ (which is not obvious how to define for masked modeling [2]), one can show that during training the model learns to minimize the KL divergence between the distribution of predictions of masked residues $\sum_{i \in M}{\log{p(\hat{ c_{\it{i}}} = c_{\it{i}} \vert {\bf c_{\setminus \it{M}}})}}$ and the true joint conditional distribution $\log{q(\hat{{\bf c_{\it{M}}}} = {\bf c_{\it{M}}} \vert {\bf c_{\setminus \it{M}}})}$. Please see [2 (Appendix A)] for the derivation. This is an interesting point and we will discuss it in the revised paper.

Please note that we slightly adapted the notation of the reviewer to match the rest of the formulas in our paper. We use $\log{p(\hat{{\bf c_{\it{M}}}} = {\bf c_{\it{M}}} \vert {\bf c_{\setminus \it{M}}})}$ instead of $\log{p(\hat{M} = M \vert \mathbf{c}_{\backslash M})}$, as the $M$ denotes a set of masked nodes, and $\mathbf{c}_M$ their values.

Reviewer comment (Experiments):
When considering the data, it's important to note that the DIPS dataset may not have broad applicability for Protein-Protein Interaction (PPI) tasks. This is because PPI tasks involve not only binding, as observed in DIPS, but also encompass other aspects such as reaction, Ptmod, and activation labels, as described in reference [1].

Author response:
We appreciate this important comment and agree that DIPS (as well as Protein Data Bank in general) does not extend beyond protein binding. Please, however, note that the focus of our paper is on designing protein-protein interactions for higher binding energy, and we do not study other, indirect ways of cooperation and mutual regulation between proteins. In this setup, the DIPS dataset (as well as our improved PPIRef) provides an important resource for self-supervised pre-training. Nevertheless, we believe that the reviewer’s comment is very stimulating for potential future work. Extending PPIformer to also learn from PPI networks, where nodes are proteins and edges are interaction types, may lead to improved performance, inspired by [1, 2]. We will cite and discuss the suggested reference.

Reviewer comment (Reproducibility):
In the optimization of a human antibody against SARS-CoV-2 and engineering staphylokinase for enhanced thrombolytic activity, how were the mutation pools obtained?

Author response:
For the SARS-CoV-2 test set, the authors of [3] selected a pool of all 494 possible single-point mutations on the heavy chain CDR region of the antibody. Out of these, five mutations were labeled as positive based on wet-lab experiments (please see [3] for the details on experimental setup). We obtained all the data from the official [RDE-PPI repository](https://github.com/luost26/RDE-PPI/tree/main).

For the staphylokinase test set, we used the results of [4]. The authors of [4] measured the effects of mutations of staphylokinase on its thrombolytic activity. 80 of these annotated mutations are located at its interface with microplasmin, where 28 mutations reportedly increased the protein's thrombolytic activity, while the remaining 52 decreased it. We use these mutations as an another independent test set. Please see Appendix B.2 "Test datasets" in the updated paper for more details.

Reviewer comment (Experiments):
Can you explain the process used to identify the five PPI outliers mentioned in Datasets?

Author response:
We select PPI outliers based on their similarity as defined by the authors of the SKEMPI v2.0 dataset [5]. Specifically, PPIs are considered similar if they share an interacting partner or a homologous binding site. Then, the outlier PPIs are defined as PPIs that do not have other similar PPIs. These outliers are depicted in Figure 2 of the original SKEMPI v2.0 publication [5] as connected components of two nodes (proteins), along with several other connected components mentioned in footnote 2 (on page 19) of our updated manuscript. 

We identified five outlier PPIs by manually analyzing these examples from the figure, aiming to select those with both positive and negative ddG annotations. We succeded at selecting the five outliers only after examining the vast majority of candidates, which highlights the difficulty of choosing a reasonable test set from highly biased SKEMPI v2.0. Lastly, we confirmed that the selected PPIs do not have structural homologs among the other (training) PPIs by calculating interface similarities with iAlign. For more details, please see Appendix B.2 "Test datasets" in the updated paper.

Reviewer comment (Experiments):
IDIST could significantly improve the computation time needed to assess dataset redundancy, which can make redundancy tests a part of every dataset curation. However, I believe that establishing IDIST as part of dataset curation, requires more validation than comparing it against iAlign and for 100 pdbs. You should compare with more methods (eg Tm-align and US align) and repeat it for more pdbs (especially because you define the threshold of IDIST from the iAligh comparison)

Author response:
We appreciate the reviewer’s idea to establish iDist as a standard tool for dataset curation. We have performed a preliminary experiment according to the reviewer’s suggestion and demonstrate that iDist accurately approximates USalign as well (details below, and illustrated in the new supplementary **Figure 7** in the updated version of the manuscript). We plan to extend the benchmark to more than 100 PDB entries (2,7M PPI pairs) and potentially other methods for the camera-ready version of the paper. It was hard to do a large-scale analysis in the limited time of the rebuttal. Please note that we have not benchmarked iDist against TM-align, as suggested, because iAlign is an adaptation of TM-align to the domain of PPIs developed by the authors of TM-align (i.e. the group of Jeffrey Skolnick). Hence, we expect iAlign is better suited for PPIs than TM-align. 

In detail, iDist achieves high precision (99%) and recall (97%) with respect to near-duplicate PPI structures detected using USalign. Notably, this is a similar level of performance we achieved when benchmarking our iDist against iAlign. However, the iAlign and USalign methods modify TM-align differently, and with different constants, which introduces a shift in the marginal distributions of their alignment scores. We found that the threshold of 0.7 we used for the iAlign score corresponds approximately to a threshold of 0.8 on the USalign score (the threshold of iDist does not need to be adjusted for accurate approximation; see Figure 7 in the supplementary material). Regarding the computational time, we find that iDist is roughly 100 times faster than USalign, confirming the need for iDist for large-scale PPI analysis.

Reviewer comment (Presentation):
Appendix Figure 3/B: You mistakenly have the same pdb code in both subfigures.

Author response:
Please note that this is not a mistake. Both PPI interfaces (E-M and N-D) come from the same PDB entry with the code [2V6A](https://www.rcsb.org/3d-view/2V6A). The complex is a 16-mer composed of two unique proteins. It is important to analyze the composition of interfaces within a single complex because symmetric complexes introduce a lot of redundancy (see also the caption under Figure 1 in the updated paper, which has been rewritten to be clearer).

Reviewer comment (Experiments):
I am worried that the F1 features introduce a data leakage and I would like to clarify it a bit: Dauparas et al. describe utilizing distances between N, Cα, C, O, and a virtual Cβ placed based on the other backbone atoms, but their down-streaming task is predicting protein sequences in an autoregressive manner from the N to C terminus. In your case, you mask a few amino acids, so with high probability neighbors of a masked residue will not be corrupted. So I am worried that the F1 features can provide information regarding the "available space" in a certain position, biasing the model towards specific amino acids that fit well the available space. Is that a concern of yours?

Author response:
We appreciate the reviewer's detailed analysis of our featurization. We indeed put high emphasis on designing F1 features such that they do not introduce data leakage.

The studied training objective is to predict masked residues. In other words, our labels are the types of masked amino acids. The F1 features are independent of any specific amino acid types and solely capture the geometric properties of the backbones of interacting proteins. As the reviewer correctly describes, these features only consider the distances and angles between backbone atoms, being agnostic to side chains. Therefore, the F1 features do not introduce data leakage as they are agnostic to the labels (i.e. types of amino acids).

Instead of introducing leakage, the F1 features provide a conditioning for predicting amino acids. Our rationale is that in protein interface engineering, where the goal is to enhance the affinity for a partner, it is typically desirable to increase binding energy while preserving other protein properties such as stability. Conditioning the model on a fixed backbone (and hence fold) can potentially lead to more successful design of protein binders. Therefore, the F1 features condition the predictions on the "available space" in terms of the backbone. However, it is important to note that the "available space" in terms of side chains is implicitly modeled to be flexible, in order to capture the diversity of possible conformations upon mutations. This is because we only capture the backbone geometry (coordinates X along with F1 features) and amino acid types (F0 features), rather than the precise atomic geometry of side chains or their chi angles. For example, consider a protein fragment consisting of three amino acids: glutamate-[MASK]-glutamate. While the prediction for the masked residue is constrained by the backbone through the F1 features, it is not constrained by the available space between side chains in the wild-type structure. Instead, the model can implicitly capture the flexibility of chi angles since it “does not see” the coordinates of glutamate side-chain atoms but only “knows” the types of the amino acids.

Reviewer comment (Experiments):
Would be nice if you could also compare your methods with baselines on the normal PDB-disjoint split, as this is also useful in real world cases. If PPIformer also performs best, the results would be more consistent and persuasive.

Author response:
We appreciate the reviewer's comment, but we would like to again emphasize that the PDB-disjoint splitting of SKEMPI v2.0 does not reflect the desired generalization in real-world scenarios. For instance, the first two PDB interaction codes present in the dataset, [1CSE_E_I](https://life.bsc.es/pid/skempi2/database/browse/mutations?keywords=mutations.pdb+contains+%221CSE_E_I%22+or+mutations.protein_1+contains+%221CSE_E_I%22+or+mutations.protein_2+contains+%221CSE_E_I%22&protein_search=1CSE_E_I) and [1ACB_E_I](https://life.bsc.es/pid/skempi2/database/browse/mutations?keywords=mutations.pdb+contains+%221ACB_E_I%22+or+mutations.protein_1+contains+%221ACB_E_I%22+or+mutations.protein_2+contains+%221ACB_E_I%22&protein_search=1ACB_E_I), come from different PDB entries ([1CSE](https://www.rcsb.org/structure/1CSE) and [1ACB](https://www.rcsb.org/structure/1ACB)) but have nearly identical 3D structures and sequences. Additionally, they are both labeled with the same six mutations, with slightly different ddG values. Despite this, they are divided into separate folds in the PDB-disjoint split by [5], exemplifying the data leakage problem. In SKEMPI v2.0, there are three times more unique PDB codes than PPIs, leading to most test PPIs having a near-duplicate PPI in the training set. Additionally, every third test mutation is located on a PPI present in the training set (please see Appendix B.2 "Test Datasets" for details).

Following the request of the reviewer, we have now re-trained our identical PPIformer architecture in the RDE-Net setup and examine its performance below. We would like to emphasize that for our method we use the same hyperparameters (e.g. architecture, loss and data representation) and do not tune the hyperparameters for these leaked splits. We only adapt the batch size from 32 to 8 (we also experimented with different batch sizes when training RDE-Net on our splits). Overall, PPIformer has competitive performance. More specifically, PPIformer performs reasonably on the overall evaluation (although it is not the best method, it often is the second best). In addition, PPIformer is the best method according to some of the metrics on the challenging set-up of multi-point mutations, which may better reflect its generalization capabilities. Please also see the [review by Sjjh](https://openreview.net/forum?id=xcMmebCT7s&noteId=bfXV4RNeg5) highlighting the importance of multi-point mutations and our corresponding response for some other details.

Overall performance:
|          |                | Per-Structure |           | Overall |           |        |        |        |
|----------|----------------|---------|----------|---------|----------|--------|--------|--------|
| **Category** | **Method**         | **Pearson** | **Spearman** | **Pearson** | **Spearman** | **RMSE**   | **MAE**    | **AUROC**  |
| Sequence Based | ESM-1v        | 0.0073  | -0.0118  | 0.1921  | 0.1572   | 1.9609 | 1.3683 | 0.5414 |
|              | PSSM           | 0.0826  | 0.0822   | 0.0159  | 0.0666   | 1.9978 | 1.3895 | 0.5260 |
|              | MSA Transf.    | 0.1031  | 0.0868   | 0.1173  | 0.1313   | 1.9835 | 1.3816 | 0.5768 |
|              | Tranception    | 0.1348  | 0.1236   | 0.1141  | 0.1402   | 2.0382 | 1.3883 | 0.5885 |
| Energy Function | Rosetta       | 0.3284  | 0.2988   | 0.3113  | 0.3468   | 1.6173 | 1.1311 | 0.6562 |
|                | FoldX         | 0.3789  | 0.3693   | 0.3120  | 0.4071   | 1.9080 | 1.3089 | 0.6582 |
| Supervised   | DDGPred        | 0.3750  | 0.3407   | **0.6580** | 0.4687   | **1.4998** | **1.0821** | 0.6992 |
|              | End-to-End     | 0.3873  | 0.3587   | 0.6373  | 0.4882   | 1.6198 | 1.1761 | 0.7172 |
| Unsup./Semisup. | B-factor    | 0.2042  | 0.1686   | 0.2390  | 0.2625   | 2.0411 | 1.4402 | 0.6044 |
|                | ESM-IF        | 0.2241  | 0.2019   | 0.3194  | 0.2806   | 1.8860 | 1.2857 | 0.5899 |
|                | MIF-Dlogit    | 0.1585  | 0.1166   | 0.2918  | 0.2192   | 1.9092 | 1.3301 | 0.5749 |
|                | MIF-Net.      | 0.3965  | 0.3509   | 0.6523  | 0.5134   | 1.5932 | 1.1469 | 0.7329 |
| RDE-PPI      | RDE-Linear     | 0.2903  | 0.2632   | 0.4185  | 0.3514   | 1.7832 | 1.2159 | 0.6059 |
|              | RDE-Net.       | **0.4448** | **0.4010** | 0.6447  | **0.5584** | $\underline{1.5799}$ | $\underline{1.1123}$ | **0.7454** |
| Ours         | PPIformer      | $\underline{0.4281}$ | $\underline{0.3995}$ | $\underline{0.6450}$ | $\underline{0.5304}$ | 1.6420 | 1.1186 | $\underline{0.7380}$ |

Reviewer comment (Experiments):
What is the logic behind your architecture choice? What is the size (#params) of your model?

Author response:
The key logic behind our architecture is to maximize generalization of the model to unseen diversity of protein-protein interactions (PPIs). Our goal is to capture interaction patterns in potentially flexible and mutated protein-protein interfaces. To achieve this, we have chosen an architecture capable of (i) capturing coarse-grain representations of PPIs to implicitly model side-chain flexibility, (ii) effective pre-training from large unlabeled data, and (iii) seamless fine-tuning on small labeled data. The encoder f, represented by Equiformer, effectively achieves (i). The combination of the classifier g and masked modeling enables (ii). Finally, the log odds ratio enables (iii) without introducing any additional parameters (see Equation 8).

Our model has 12,333,366 parameters.

Reviewer comment (Experiments):
For real-world case studies, I suggest you try your model on deep mutational scanning (DMS) data, where the enrichment ratios of all single-point mutations are measured (though with higher noise). Ranking mutations on a single protein where only a few mutations have been experimentally measured is less convincing IMHO.

Author response:
We agree with this valuable comment. While preparing the paper, we explored this direction but were unable to find larger and high-quality datasets of labeled protein-protein interaction variants. For instance, the ProteinGym benchmark includes 87 DMS assays, where the experimentally measured fitness is complex and, therefore, treated as a black box [2]. This means that without a detailed manual analysis of each assay, it is difficult to identify assays that are directly representative of  protein-protein interactions. Furthermore, even if relevant assays are found, it is unclear whether the corresponding interactions have solved 3D structures, hampering the application of structure-based methods (such as ours). Nevertheless, while we cannot compare ddG predictors on DMS data, we compare the best available predictor trained on DMS from ProteinGym data on the task of ddG prediction. We find that it performs worse than our method (please see our comment on the Weakness 2 by the [hYdu reviewer](https://openreview.net/forum?id=xcMmebCT7s&noteId=0tshmegDww) for the details).

Additionally, previous work has already attempted to evaluate ddG predictors using deep sequencing data. Specifically, the DGCddG model was evaluated on the T56 target from the CAPRI ROUND 26 benchmark from 2012 [3]. The benchmarked complex includes 855 annotated mutations, and DGCddG achieved a near-zero correlation. The poor performance could be due to either the low quality of the data or poor generalization of the model (which is not state-of-the-art and is overfitted to a random split of mutations). Considering the near-zero correlation of all other evaluated methods strongly suggests the validity of the first point. We hypothesize that the measured fitness for T56 may not necessarily depend on the affinity of the studied complex, and the computationally constructed structure may have low quality.

In conclusion, we find that currently available deep mutation scanning data is not suitable for training and evaluating the effects of mutations on protein-protein interactions. However, we believe that curating such datasets is a highly promising direction for the next generation of ddG predictors, especially considering recent advances in high-throughput assays [4]. If the reviewer is aware of any relevant work that we may have missed, we would be very grateful for providing us with the reference. We will incorporate it in the paper.

**References**

[1] Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018).

[2] Notin, Pascal, et al. "Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval." International Conference on Machine Learning. PMLR, 2022.

[3] Jiang, Yelu, et al. "DGCddG: Deep Graph Convolution for Predicting Protein-Protein Binding Affinity Changes Upon Mutations." IEEE/ACM Transactions on Computational Biology and Bioinformatics (2023).

[4] Baryshev, Alexander, et al. "Massively parallel protein-protein interaction measurement by sequencing (MP3-seq) enables rapid screening of protein heterodimers." bioRxiv (2023): 2023-02.

[5] Luo, Shitong, et al. "Rotamer Density Estimator is an Unsupervised Learner of the Effect of Mutations on Protein-Protein Interaction." bioRxiv (2023): 2023-02.

Reviewer comment (Experiments):
Evaluations on a broader range of datasets are preferred in terms of PPI and mutational effect predictions. Currently, all the test samples are selected from a larger benchmark dataset, making it skeptical that the results might be cherry-picked.

Author response:
Thank you for this important comment. Please note that in our work we use three independent test datasets (see Section 5.1 Datasets). First, we select 5 outlier (i.e. not having a training homologue) test PPIs from the SKEMPI v2.0 dataset. SKEMPI v2.0 encompasses the vast majority of published experimental ddG studies. This dataset, indeed, represents a selection from a larger benchmark dataset. We selected the test samples once to evaluate the generalization of our model before seeing the results. We did not overfit our method to the test set by exploring any other subsets. Please see Appendix B.2 “Test Datasets”, which now describes in detail the construction of the SKEMPI v2.0 test set.

Moreover, we evaluate our method on two additional (and separate) datasets, which are not part of SKEMPI v2.0 and come from independent studies. The “Optimization of a human antibody against SARS-CoV-2” provides an independent evaluation on the benchmark proposed by [1] and [2], the studies proposing DDGPred and RDE-Network (ICLR 2023). Finally, to compare generalization of different methods even more robustly, we curated a novel independent test set for the problem of  “Engineering staphylokinase for enhanced thrombolytic activity”. This dataset was mined from [12].

We hope that our test setup involving three mutually independent test sets provides enough evidence that the results are not cherry-picked. Please also see our comment on attempts to obtain more test data in the answer to the [ZbAC reviewer’s Question 4](https://openreview.net/forum?id=xcMmebCT7s&noteId=jfNTtLMy5P). Overall, it is challenging to find additional independent and high-quality test data.

Reviewer comment (Experiments):
Comparison of baseline methods is limited. At least some other deep learning methods should be included. For instance, for mutational effect prediction, the authors might refer to https://github.com/OATML-Markslab/ProteinGym.

Author response:
Thank you for your important comment. Please note that, while the number of evaluated baselines is not excessive, we have initially selected methods among the most representative methods in four available categories:
1. Physics-based simulators. Flex ddG is the most performant and elaborate force-field-based protocol for binding ddG prediction.
2. Evolutionary methods. MSA-Transformer is a well-established method that might benefit from using multiple sequence alignment (MSA) compared to other baselines (MSA is crucial, for example, for AlphaFold2).
3. Unsupervised models. ESM-IF is a state-of-the-art inverse folding method, trained on the extensive AlphaFold DB. It was shown to effectively score single-point mutations from SKEMPI v2.0.
4. Supervised models. RDE-Network is a recent, high-quality, and state-of-the-art method fine-tuned on SKEMPI v2.0.

We value the suggestion to consider additional baselines from ProteinGym. Consequently, we have now added the evaluation of GEMME, the best publicly available model in [ProteinGym](https://www.proteingym.org/substitutions) (GEMME ranks second among 34 methods and is publicly available, unlike the top-ranked model, TranceptEVE). We find that GEMME performs worse on most metrics than our PPIformer (please refer to the table below). However, it is worth noting that GEMME, as well as other ProteinGym methods, was not trained for PPI design. Instead, it was trained to optimize the global fitness of a protein with respect to deep mutational scanning experiments.

Reviewer comment (Evaluation):
The prediction results are not reported with standard deviation, which makes it hard to tell whether the performance is statistically significant.

Author response:
We agree with this valuable comment. However, it is important to note that the currently available test data for ddG prediction is limited, and the performance of the methods varies significantly across different protein-protein interactions (see Table 6 in the paper). As a result, the traditional bootstrapping approach [6, 7] or estimating the standard deviation across PPIs are not useful due to the high variance. This may explain why statistical significance on SKEMPI predictions has  been evaluated before (to the best of our knowledge). In fact, none of the methods we found and mention in the "Related Work" section assesses statistical significance. Nonetheless, we have now evaluated the stability of our method by retraining our model using two additional random seeds and reporting the mean and standard deviation of the test results. We provide the updated table 2 from the main paper below:

| Category                                  | Method                   | Spearman $\uparrow$ | Pearson $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | ROC AUC $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ |
|-------------------------------------------|--------------------------|----------------------|--------------------|-----------------------|-------------------|---------------------|-------------------|---------------------|
| Force field simulations                   | Flex ddG        | $0.54$               | $0.57$             | $0.63$                | $0.62$             | $0.84$              | $1.60$            | $2.00$              |
|                                           |                          |                      |                    |                       |                   |                     |                   |                     |
| Machine learning                          | GEMME           | $\underline{0.38}$     | 0.41               | $\textbf{0.60}$         | 0.49              | 0.74                | 2.16              | 2.81                |
| Machine learning | MSA Transformer | $0.37$               | $\underline{0.45}$   | $0.51$                | $0.38$            | $\underline{0.76}$   | $5.99$            | $6.77$              |
| Machine learning | ESM-IF          | $0.32$               | $0.31$             | $0.36$                | $0.28$            | $0.69$              | $1.84$            | $2.11$              |
| Machine learning | RDE-Net.        | $0.24$               | $0.30$             | $\underline{0.54}$      | $\textbf{0.65}$     | $0.67$              | $\underline{1.70}$  | $\underline{2.02}$    |
| Machine learning | PPIformer (ours)| $\textbf{0.44} \pm 0.023$ | $\textbf{0.46} \pm 0.029$ | $\textbf{0.60} \pm 0.014$ | $\underline{0.61} \pm 0.012$ | $\textbf{0.78} \pm 0.018$ | $\textbf{1.64} \pm 0.011$ | $\textbf{1.94} \pm 0.005$ |

The test performance for the individual training runs is provided below. Seed #1 corresponds to the originally reported performance of PPIformer.

| Method                   | Spearman $\uparrow$ | Pearson $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | ROC AUC $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ |
|--------------------------|----------------------|--------------------|-----------------------|-------------------|---------------------|-------------------|---------------------|
| PPIformer (ours)          | $0.44 \pm 0.023$               | $0.46 \pm 0.029$             | $0.60 \pm 0.014$                | $0.61 \pm 0.012$            | $0.78 \pm 0.018$              | $1.64 \pm 0.011$            | $1.94 \pm 0.005$              |
| PPIformer (ours, seed #1)          | $0.42$               | $0.46$             | $0.58$                | $0.61$            | $0.77$              | $1.64$            | $1.94$              |
| PPIformer (ours, seed #2)          | $0.47$               | $0.49$             | $0.60$                | $0.62$            | $0.80$              | $1.62$            | $1.94$              |
| PPIformer (ours, seed #3)          | $0.42$               | $0.44$             | $0.61$                | $0.60$            | $0.77$              | $1.64$            | $1.94$              |

Reviewer comment (Experiments):
Why use log odds ratio for predicting $\Delta\Delta G$, instead of training a regressor to directly predicting the $\Delta\Delta G$ values?

Author response:
We chose to use log odds ratios due to the thermodynamic motivation discussed in Section 4.4. In particular, log odds ratio offers two important advantages: (i) it introduces an inductive bias of antisymmetry (i.e. $\Delta \Delta G_{mut \rightarrow wt} = - \Delta \Delta G_{wt \rightarrow mut}$), and (ii) it enables seamless fine-tuning after log-likelihood based pre-training, without introducing any additional parameters. Point (ii) allows us to fully unlock the potential of the self-supervised pre-training. Specifically, employing log odds for $\Delta \Delta G$ prediction, the self-supervised objective can be interpreted as learning to estimate $\Delta G$ (see Section 4.4).

We validate our argument by performing two additional ablation experiments. First, following [8] and [9], we implement a baseline regressor that performs two forward passes to predict $\Delta \Delta G$: one for the wild-type structure and another for a mutated structure (i.e. with replaced  $F_0$ features capturing one-hot encoded amino acid types). Then, it estimates $\Delta \Delta G$ by taking the average of node embeddings corresponding to residues being mutated in both structures, concatenating them, and applying a simple two-layer regressor of the form: $$ 
Linear(256, 128) \rightarrow ReLU \rightarrow Linear(128, 1).
$$ This approach results in a significant drop in performance (see the first row “Naive regressor” in the table below). Second, we improve this naive design by incorporating antisymmetry: removing biases in the Linear layers, replacing ReLU with Tanh (an odd non-linear function), and subtracting averaged embeddings instead of concatenating them. This improvement leads to a performance boost compared to the naive baseline, but still strongly underperforms the log odds ratio in terms of practically important metrics (see the second row “Antisymmetric regressor” in the table below).

| Method                   | Spearman $\uparrow$ | Pearson $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | ROC AUC $\uparrow$ | MAE $\downarrow$ | RMSE $\downarrow$ |
|--------------------------|----------------------|--------------------|-----------------------|-------------------|---------------------|-------------------|---------------------|
| PPIformer (Naive regressor)          | $0.298$               | $0.330$             | $0.204$                | $0.258$            | $0.724$              | $1.844$            | $2.210$              |
| PPIformer (Anysymmetric regressor)          | $0.348$               | $0.382$             | $0.494$                | $0.626$            | $0.658$              | $1.720$            | $1.964$              |
| PPIformer (ours)          | $0.44 \pm 0.023$               | $0.46 \pm 0.029$             | $0.60 \pm 0.014$                | $0.61 \pm 0.012$            | $0.78 \pm 0.018$              | $1.64 \pm 0.011$            | $1.94 \pm 0.005$