Paper: Split and Merge Proxy: pre-training protein inter-chain contact prediction by mining rich information from monomer data (ICLR-2024)

Reviewer comment (Theory):
The authors did not discuss the biological insight of their proposed pre-training method. There is no good biological sense that the interaction between two parts of a monomer could be useful to reveal the true nature of the interaction among chains of multimers.

Author response:
There are mainly two aspects of biology insight about our proposed method, and we illustrate the advantages of the interaction between two parts of a monomer (pseudo data) from the perspectives of **Richer Monomer Patterns** and **Local Part Relationships**.
>
> **Richer Monomer Patterns:** The pseudo data contains a wealth of monomer patterns, which can be highly beneficial for the multimer model. This can be particularly true in the case of domain-domain interactions in monomers. Studies have shown that these interactions bear similarities to the chain-chain interactions that occur in multimers [1]. Hence, the diversity and complexity of these monomer patterns can provide valuable insights for predicting multimer interactions and structures.
> **Local Part Relationships:** Even though a random split scheme may not accurately depict the global physical relationships between two pseudo chains, it can still shed light on the local part relationships. Traditional methods for multimer contact prediction and structure prediction often compute the structure or contact based on local parts across two chains [2][3]. Therefore, despite its limitations, the random split scheme can still provide useful local patterns that can aid the computational model in predicting multimer structures.

Reviewer comment (Experiments):
λ the threshold for pseudo label and R the range for random split are important hyperparameters. I hope the authors could further discuss how their different values impact the pre-training process.

Author response:
Thanks for your suggestion. The range for random R can be regarded as R={$λ_1$~ $λ_2$}. So we can only focus on R and discuss different ranges on the split site as follows.
>
>**For R the range for random split:** Splitting at 1/5-4/5 would cause some too-short chains which are trivial and will be filtered out by the DeepInteract pre-processing step (DeepInteract requires that every single chain must have at least 20 residues). Further, splitting at other locations may lead to similar results. In the original paper, the ablation study about such splitting range shown in Table 6 is operated at the test set, which amplified the performance gap between different experiment groups because the scale of the test set (32 samples) is small. Here, we provide more stable results on the validation set (3548 samples) as the following table, shows that splitting at other locations except 1/5-4/5 would lead to similar results but 1/3-2/3 is still the best.
>The different split range results on the DIPS-Plus validation set are shown as follows:
>| Range   | P@L/10 | P@L/5 | P@L/2 | P@L   | R@L   | R@L/2 | R@L/5 | R@L/10 |
>|---------|--------|-------|-------|-------|-------|-------|-------|--------|
>| 2/5~3/5 | 46.57  | 43.62 | 36.42 | 28.16 | 41.89 | 28.91 | 14.72 | 7.84   |
>| 1/3~2/3 | 49.82  | 46.44 | 38.02 | 28.71 | 43.28 | 30.63 | 15.84 | 8.49   |
>| 1/4~3/4 | 49.14  | 45.66 | 37.25 | 28.25 | 42.07 | 29.58 | 15.41 | 8.27   |
>| 1/5~4/5 | 41.90  | 39.08 | 32.34 | 24.75 | 37.53 | 25.95 | 13.30 | 7.08   |

Reviewer comment (Experiments):
Data Quality: While mining information from monomer data might provide more data for multimer contact prediction, the synthesized data might not be as accurate as genuine multimeric structural data. This could lead to instability in model training or inaccuracies in prediction.

Author response:
Although the current pseudo multimer data includes several noises, the pros outweigh the cons. Data from monomers may not be as good as true multimers, but the gain from monomer data has been demonstrated on various datasets. The main reason is that the deep learning model can neglect a certain level of data noise [1]. Since there is less real multimer data and a lot of monomer data, there is a great potential to bring better gains by improving multimer predictions with monomer data.

Reviewer comment (Theory):
Challenges with Transfer Learning: While pre-training methods like SMP can leverage monomer data, the transfer from monomers to multimers might not always be seamless. This means that features in monomer data might not entirely correspond with those in multimer data.

Author response:
We agree that there is a seam between monomer and multimer. However, this seam is not quite large. Some paper has demonstrated their potential consistency [2] by directly utilizing the monomer contact prediction model in multimer scenarios. Similar results can also be found in the first row of Table 5 in our original paper, showing that the monomer model can directly show the ability to treat the multimer model. So with tranferling, our model can achieve satisfactory results in existing benchmarks.

Reviewer comment (Experiments):
Issues with Experimental Validation: Due to the relative scarcity of genuine multimeric structural data, it might be challenging to adequately validate the model's performance. This might lead to overfitting or an over-reliance on synthesized data.

Author response:
We think your concerns are reasonable, but experiments on a large number of multimer data show that our method works well. We previously considered whether we might be overfitting on a multimer dataset, so we evaluated it on the CASP-CAPRI and DB5 datasets in addition to the DIPS-plus dataset, which all showed the effectiveness of our method.

Reviewer comment (Experiments):
Structural Diversity: The diversity of protein multimeric structures might exceed what monomer data can provide, possibly limiting the model's generalization capability.

Author response:
We agree with that the monomer diversity is lower than the multimer, but we believe this will not limit the generalization of our model. The experiments in the first line of Table 5 show that only using monomer data cannot lead to a satisfied multimer model. So in our implementation, the monomer data is used for pre-training and multimer data is combined and used to fine-tine. So the information provided by the monomer is additional and does not limit the generalization capability. Actually, we have evaluated the model's generalization capability on the CASP-CAPRI and DB5. It shows our cross-set generalization capability.

Reviewer comment (Theory):
Other Biological Limitations: Biologically, not all monomers can simply be split and merged to simulate multimeric structures. Some proteins are highly specific in structure and function, which might impact prediction accuracy.

Author response:
Following biological limitations will lead to extremely small-scale data that cannot introduce satisfactory data for pretraining. Random splitting may introduce biological noise but introduce large-scale data, leading to better pre-training results. In detail, we have tried to split at the protein domain level. However, the domain-level sub-parts only occur in long protein sequences. We do statistics on ~2500 monomers in PDB dataset and find that only 5 proteins have domain sub-parts and can be split at the domain level. Such less extra data cannot benefit network pre-training. However, we find that we can simply split and merge monomers to improve multimer prediction performance.

Reviewer comment (Experiments):
Would this approach also work for antibody CDR design?

Author response:
Yes. Thanks for your discussion, we think that we can use our approach for antibody CDR design.  In the task of antibody CDR structure design, heavy chain and light chain interact to form a complex antibody structure, similar to the structure prediction of dimers, and the multimer structure prediction model has the potential to work in this task [6]. We can use a large number of monomers to obtain pseudo-multimers to help improve the performance of heavy chain and light chain interaction prediction. In the future, we will extend our approach to antibody CDR design.

Reviewer comment (Evaluation):
DB5 is the most appropriate benchmark dataset, as it provides the individual chains in their unbound state. In my opinion, using DIPS-plus data for testing is not appropriate, since the unbound structures are not available. It is a highly valuable resource for training and validation. Your results demonstrate how much more difficult this task is, and makes me question the value of the evaluation over the DIPS-plus and CASP-CAPRI datasets. It also demonstrates that we are still very far from being able to accurately predict inter-chain contacts in a realistic scenario.

Author response:
We believe that all these benchmarks are meaningful for the inter-chain contact prediction task. Results on DIPS-plus and CASP-CAPRI show fair comparisons between our SMP and other state-of-the-art methods. Although these results cannot reflect whether the model can be applied in real scenarios or not, they still demonstrate the superiority of our model. We agree that results on DB5 show that this topic is far away from real application because of its extreme challenge. However, we still achieve a new state of the art on this dataset and plan to continue to solve that topic to make it close to real application in the future.

Reviewer comment (Experiments):
There is a potential concern for information leakage if the monomer data contains structures that are similar to the trained complexes. I don't think that is a concern, since the monomer data does not contain information about the labels; in fact, I would be curious to know whether allowing the model to learn from the monomers in a complex helps improve performance.

Author response:
The monomer data would benefit the multimer models for the following reasons:

**Richer Monomer Patterns:** The pseudo data contains a wealth of monomer patterns which can be highly beneficial for the multimer model. This can be particularly true in the case of domain-domain interactions in monomers. Studies have shown that these interactions bear similarities to the chain-chain interactions that occur in multimers [1]. Hence, the diversity and complexity of these monomer patterns can provide valuable insights for predicting multimer interactions and structures.

**Local Part Relationships:** Even though a random split scheme may not accurately depict the global physical relationships between two pseudo chains, it can still shed light on the local part relationships. Traditional methods for multimer contact prediction and structure prediction often compute the structure or contact based on local parts across two chains [2,3]. Therefore, despite its limitations, the random split scheme can still provide useful local patterns that can aid the computational model in predicting multimer structures.

Reviewer comment (Evaluation):
The performance advantage for EquiDock is unclear, as the results using SMP are slightly worse than the EquiDock published results, but comparable to the result when they ran it.

Author response:
The EQUIDOCK we reported in the original paper is our reproduced results with the EQUIDOCK official GitHub repository with the default setting. The reproduced results have a litter drop with the paper-reported one because of the device change, which is common in machine learning reproduction. We compare our SMP and EQUIDOCK under the same setting and device for fairness and to achieve better results. 

Further, to demonstrate the superiority of our SMP, we conduct the experiments in a series of less training data scenarios on the DIPS. We compare the SMP with EQUIDOCK in 40%, 30%, 20%, and 10% DIPS training data. The results are as follows:
||  |         |        |     DIPS   |             |        |        |
|:-----------------:|:-----------------:|:-------:|:------:|:------:|:-----------:|:------:|:------:|
||  |         |  Complex RMSD       |        |             |    Interface RMSD    |        |
|Data ratio|Method |Median| Mean| Std|Median | Mean| Std|
|0.4|        EQUIDOCK        |  15.11    | 16.34   |  7.16  |      12.44   |13.37  |  6.56   |
||        SMP        |  14.78  | 15.06  | 6.72  |    11.20    | 12.59 |  6.41  |
|0.3|        EQUIDOCK        |    16.20   | 16.87   | 7.06   |       12.15  | 13.64 |  6.86   |
||        SMP        |  15.79   |  15.40 |  7.47 |   11.55     |  12.25|  6.77  |
|0.2|        EQUIDOCK        |    17.03   | 17.08 | 7.80  |   13.65      | 14.99 |  7.47  |
||        SMP        |   15.45   | 16.15  |  8.17 |   11.44    | 13.04 |  6.85  |
|0.1|        EQUIDOCK        |   18.21    | 18.27 |  6.66 | 14.73        | 15.50 |  6.67  |
||        SMP        |   14.55   | 16.17  | 7.22 |   11.72    |  13.62|   6.57 |

The experiments show that our SMP can adapt to different volume-level training data situations, having the potential to save the extra cost of collecting multimer data.

Reviewer comment (Evaluation):
"However, due to the small scale of existing multimer data, current models are less accurate in protein inter-chain contact prediction." Less accurate than existing methods? AlphaFold-Multimer will definitely be less accurate than its monomeric counterpart as complex structure prediction is a more difficult problem.

Author response:
Sorry for your confusion, we want to express that current models have low performance in the inter-chain contact prediction task because of the small-scale of existing multimer data.

Reviewer comment (Writing):
"We all use HHBlits (Remmert et al., 2012) with Uniclust30 (Mirdita et al., 2017) database for MSA, and PSAIA (Mihel et al., 2008) to calculate geometric features." What do you mean by geometric features?

Author response:
We compute geometric features as same with the DeepInteract [4]. The geometric features include inter-residue distances, directions, and orientations with the angles between each residue pair’s amide plane normal vectors.

Reviewer comment (Evaluation):
Please define the metric you use (e.g. P @ L/5). Are those numbers in percentages?

Author response:
Yes, they are in percentages. We use the same metrics as shown in the DeepInteract [4]. L means the length of shortest chain for a protein complex, `P @ L/5` represents the model’s top-L/5 precision.

Reviewer comment (Experiments):
"After filtering extreme data, such as too long, too short sequences and high relative data with other datasets," please define "high relative data..." Did you mean sequence similarity to other datasets? The threshold used to define that is important. Also, I can understand why you wouldn't want to include very short sequences, but why remove long ones?

Author response:
Yes, "high relative data" means that sequence similarity to other datasets. We follow DeepInteract [4] to set the threshold 30% to filter complex among our test sets and training and validation sets. The threshold of 30% sequence identity is a commonly used criterion in bioinformatics for assessing the similarity between protein sequences [5,6]. This definition will be updated in the final revision.

Reviewer comment (Reproducibility):
"Filtering the overlap between the original CASP-CAPRI data and the DIPS-Plus." Please explain how overlap was computed.

Author response:
We follow DeepInteract [4] and filter very long sequences because they will cause severe compute consumption or even make the GPUs out of memory. We also follow DeepInteract and use the same filtered CASP-CAPRI data.

Reviewer comment (Presentation):
Figure 3 is very difficult to read.

Author response:
Sorry for your confusion, we will change the size to make it clear in the final revision.

Reviewer comment (Writing):
heterologous multimer --> heteromultimer

Author response:
Thanks for your suggestion. We will update these unclear descriptions in the final revision.

Reviewer comment (Novelty):
Significance: Since AF2-Multimer and DiffDock-PP (which, by the way, should be compared in the docking benchmark) can already predict the structure of the protein complex quite well, the role of contact prediction becomes less significant. Quite similarly, in monomer structure prediction, initially contact maps are predicted and used to refine the final structure, until models like AF2 can predict protein structure end-to-end.

Author response:
Q3: Could you elaborate the significance of inter-chain contact prediction? Since AF2-Multimer and DiffDock-PP can already predict the structure of the protein complex quite well, the role of contact prediction becomes less significant. Quite similarly, in monomer structure prediction, initially contact maps are predicted and used to refine the final structure, until models like AF2 can predict protein structure end-to-end.

> 
> We clarify that the importance of contact prediction should not be neglected because: 
> 1. Structure predicted by AF-Multimer is still not too accurate to meet real applications [2,3]. Inter-chain contact prediction is still useful for further accurate structure prediction, as mentioned in [4,5]. Some recent works [6,7,8] still tried to predict the multimer structure by a two-stage approach, achieving accurate results. So we believe inter-chain contact prediction does not conflict with milestones achieved by AF series. 
>
> 2. In some scenarios, biologists may not care bout the detailed 3D structures of proteins but require the binding site and functional residues [9,10]. Thus, accurate inter-chain contact prediction can help them quickly analyze interactions between chains in a protein complex. 
> 


Reference

[2] Zhu W, Shenoy A, Kundrotas P, et al. Evaluation of AlphaFold-Multimer prediction on multi-chain protein complexes[J]. Bioinformatics, 2023, 39(7): btad424.

[3] Yin R, Feng B Y, Varshney A, et al. Benchmarking AlphaFold for protein complex modeling reveals accuracy determinants[J]. Protein Science, 2022, 31(8): e4379.

[4] Lin P, Yan Y, Huang S Y. DeepHomo2. 0: improved protein–protein contact prediction of homodimers by transformer-enhanced deep learning[J]. Briefings in Bioinformatics, 2023, 24(1): bbac499.

[5] Yan Y, Huang S Y. Accurate prediction of inter-protein residue–residue contacts for homo-oligomeric protein complexes[J]. Briefings in bioinformatics, 2021, 22(5): bbab038.

[6] Soltanikazemi E, Roy R S, Quadir F, et al. DRLComplex: Reconstruction of protein quaternary structures using deep reinforcement learning[J]. arXiv preprint arXiv:2205.13594, 2022.

[7] Quadir F, Roy R S, Soltanikazemi E, et al. DeepComplex: a web server of predicting protein complex structures by deep learning inter-chain contact prediction and distance-based modelling[J]. Frontiers in Molecular Biosciences, 2021, 8: 716973.
[8] Baek M, Anishchenko I, Park H, et al. Protein oligomer modeling guided by predicted interchain contacts in CASP14[J]. Proteins: Structure, Function, and Bioinformatics, 2021, 89(12): 1824-1833.

[9] Christopoulos A. Allosteric binding sites on cell-surface receptors: novel targets for drug discovery[J]. Nature reviews Drug discovery, 2002, 1(3): 198-210.

[10] Kagaya Y, Flannery S T, Jain A, et al. ContactPFP: protein function prediction using predicted contact information[J]. Frontiers in bioinformatics, 2022, 2: 896295.

Reviewer comment (Experiments):
When you benchmark against AF2-Multimer, are you benchmarking against the contact prediction module? Could you try computing the contact map based on the predicted structures and benchmark against that?

Author response:
Q1: When you benchmark against AF2-Multimer, are you benchmarking against the contact prediction module? Could you try computing the contact map based on the predicted structures and benchmark against that?

> Yes, we have already employed the AlphaFold series model (including AlphaFold2 (AF2) and AlphaFold-Multimer (AF-Multimer)) for contact prediction and compared them with our SMP on the DIPS-Plus, CASP-CAPRI, and DB5 test sets in the Section A in the Supplementary Material. Here, we list the comparison results again and provide further discussions.
>
>|  |         |        |     DIPS-Plus   |             |        |        |
>|:-----------------:|:-------:|:------:|:------:|:-----------:|:------:|:------:|
>|  |         |    16(Homo)    |        |             |   16(Hetero)     |        |
>|       Method      | P@ L/10 | P@ L/5 | P@ L/2 |     P@ L/10    | P@ L/5 | P@ L/2 |
>|        AF2        |   12.78   |  10.67  |  8.72  |    0.27     | 0.62 |   0.43  |
>|        SMP        |  39.81  |  33.33 |  26.02 |    20.99    | 20.07 |  15.00  |
>|  |         |        |     32(ALL proteins)   |            |        |        |
>|       Method      | P@ L/10 | P@ L/5 | P@ L/2 |     R@ L    | R@ L/2 | R@ L/5 |
>|        AF2        |  6.53    |  5.64  |  4.57  |    1.92     | 1.40 |  0.63   |
>|        SMP        |  30.40  | 26.70  |  20.51 |  24.00      | 16.02 |  8.56  |
>
>|  |         |        |     CASP-CAPRI   |             |        |        |
>|:-----------------:|:-------:|:------:|:------:|:-----------:|:------:|:------:|
>|  |         |   14(Homo)     |        |       15(Hetero)      |        |        |
>|       Method      | P@ L/10 | P@ L/5 | P@ L/2 |     P@ L/10    | P@ L/5 | P@ L/2 |
>|        AF2        |   6.34   |  3.70  |  2.32  |   0.0     | 0.0 |  0.77   |
>|        AF-Multimer        |  14.06  | 7.43  | 3.86  |    0.0    |0.0  |  0.0  |
>|        SMP        |  18.63  | 14.37  | 11.57  |  32.00     |23.49  |  18.35  |
>|  |         |        |     19(ALL proteins)   |            |        |        |
>|       Method      | P@ L/10 | P@ L/5 | P@ L/2 |     R@ L    | R@ L/2 | R@ L/5 |
>|        AF2        |  4.67   | 2.73   |  1.91  |    1.24     | 1.18 | 0.99    |
>|        AF-Multimer        |  10.36  | 5.47  | 2.85  |  2.47      | 2.31 |  2.19  |
>|        SMP        | 21.97   | 16.77  | 13.36  |  14.33     | 8.34 | 3.91   |
>
>|  |         |        |     DB5   |             |        |        |
>|:-----------------:|:-------:|:------:|:------:|:-----------:|:------:|:------:|
>|  |         |        |    55(Hetero)    |             |        |        |
>|       Method      | P@ L/10 | P@ L/5 | P@ L/2 |     R@ L    | R@ L/2 | R@ L/5 |
>|        AF2        |  0.074  |  0.074  | 0.054   |    0.17    | 0.063 |  0.063   |
>|        SMP        |  1.78  |1.88  |  1.55 |  2.53     | 1.45 | 0.69   |
>
> **Note:** We only compare AF-Multimer on CASP-CAPRI and neglect DIPS-Plus and DB5 datasets because the latter two ones have been included in the training data of AF-Multimer.
>
> **Discussions:** 
>1. The experimental results show that AF-Multimer has a better performance than AF2 in the inter-chain contact prediction task. But our SMP performs much better than both AF2 and AF-multimer. 
>2. AF2 is designed for the monomer task, so its multimer-related task cannot achieve a satisfactory result.
>3. AF-Multimer is trained on multimer data so it can deal with multimer structure prediction. Due to the limitation of the multimer data volume, AF-Multimer still achieves worse performance than our SMP.

Reviewer comment (Experiments):
Could you compare your docking results with DiffDock-PP?

Author response:
Q2: Could you compare your docking results with DiffDock-PP?

> We compare the DiffDock-PP [1] with our SMP on the DIPS test set for the docking task. The Vanilla DiffDock-PP is trained on a full DIPS training set (40,290 samples) for 170 epochs within 4 days by utilizing 8 A6000 GPUs (48GB). Currently, due to the limitation of rebuttal time and computation resources, we select the first 8,000 samples from the DIPS training set following a suggestion mentioned in a GitHub issue in DiffDock-PP official GitHub repository and keep other setting unchanged. The results are as follows:
>|  |         |   DIPS     |        |
>|:-----------------:|:-------:|:------:|:------:|
>|  |         |  Complex RMSD       |        
>|Method |Median| Mean| Std|
>|        DiffDock-PP        |  16.47    |  15.86  |  9.13  |        
>|        SMP        |  15.95  |  14.79 | 7.94  |
>
> The results show that our SMP can achieve better results than the DiffDock-PP in this scenario, which demonstrates that our SMP the has potential to help with multimer structure prediction and has the advantage in less multimer data situations. We will compare the DiffDock-PP with our SMP on the full DIPS test set in the future.

Reference:

[1] Ketata M A, Laue C, Mammadov R, et al. DiffDock-PP: Rigid Protein-Protein Docking with Diffusion Models[C]//ICLR 2023-Machine Learning for Drug Discovery workshop. 2023.

Reviewer comment (Novelty):
In response to my comment above, could you elaborate the significance of inter-chain contact prediction?

Author response:
Q3: Could you elaborate the significance of inter-chain contact prediction? Since AF2-Multimer and DiffDock-PP can already predict the structure of the protein complex quite well, the role of contact prediction becomes less significant. Quite similarly, in monomer structure prediction, initially contact maps are predicted and used to refine the final structure, until models like AF2 can predict protein structure end-to-end.

> 
> We clarify that the importance of contact prediction should not be neglected because: 
> 1. Structure predicted by AF-Multimer is still not too accurate to meet real applications [2,3]. Inter-chain contact prediction is still useful for further accurate structure prediction, as mentioned in [4,5]. Some recent works [6,7,8] still tried to predict the multimer structure by a two-stage approach, achieving accurate results. So we believe inter-chain contact prediction does not conflict with milestones achieved by AF series. 
>
> 2. In some scenarios, biologists may not care bout the detailed 3D structures of proteins but require the binding site and functional residues [9,10]. Thus, accurate inter-chain contact prediction can help them quickly analyze interactions between chains in a protein complex. 
> 


Reference

[2] Zhu W, Shenoy A, Kundrotas P, et al. Evaluation of AlphaFold-Multimer prediction on multi-chain protein complexes[J]. Bioinformatics, 2023, 39(7): btad424.

[3] Yin R, Feng B Y, Varshney A, et al. Benchmarking AlphaFold for protein complex modeling reveals accuracy determinants[J]. Protein Science, 2022, 31(8): e4379.

[4] Lin P, Yan Y, Huang S Y. DeepHomo2. 0: improved protein–protein contact prediction of homodimers by transformer-enhanced deep learning[J]. Briefings in Bioinformatics, 2023, 24(1): bbac499.

[5] Yan Y, Huang S Y. Accurate prediction of inter-protein residue–residue contacts for homo-oligomeric protein complexes[J]. Briefings in bioinformatics, 2021, 22(5): bbab038.

[6] Soltanikazemi E, Roy R S, Quadir F, et al. DRLComplex: Reconstruction of protein quaternary structures using deep reinforcement learning[J]. arXiv preprint arXiv:2205.13594, 2022.

[7] Quadir F, Roy R S, Soltanikazemi E, et al. DeepComplex: a web server of predicting protein complex structures by deep learning inter-chain contact prediction and distance-based modelling[J]. Frontiers in Molecular Biosciences, 2021, 8: 716973.
[8] Baek M, Anishchenko I, Park H, et al. Protein oligomer modeling guided by predicted interchain contacts in CASP14[J]. Proteins: Structure, Function, and Bioinformatics, 2021, 89(12): 1824-1833.

[9] Christopoulos A. Allosteric binding sites on cell-surface receptors: novel targets for drug discovery[J]. Nature reviews Drug discovery, 2002, 1(3): 198-210.

[10] Kagaya Y, Flannery S T, Jain A, et al. ContactPFP: protein function prediction using predicted contact information[J]. Frontiers in bioinformatics, 2022, 2: 896295.