Paper: EquiPocket: an E(3)-Equivariant Geometric Graph Neural Network for Ligand Binding Site Prediction (ICLR-2024)

Reviewer comment (Novelty):
Geometric aware E(3)-equivariant GNN has been applied to several very related tasks like protein-ligand docking [1]. Consequently, the novelty of introducing E(3)-equivariant GNNs may be diminished.

Author response:
Thanks for the comment. There could be a misunderstanding regarding our research task. While it's true that E(3)-equivariant GNNs are used in tasks such as protein-ligand docking (like E3bind [1]), our EquiPocket focuses on protein pocket prediction, a distinct task from docking.

Docking methods like E3Bind [1] and TankBind [2] predict protein-ligand binding poses and affinities, often using method like P2rank [4] to first identify candidate binding sites (which is also called protein pocket in our context).
**Our EquiPocket and other related works [5, 6, 7, 8, 9] share the same goal as P2rank, identifying the center of candidate binding sites on the protein rather than locating the exact interface between each ligand and the target protein. The area around this center is seen as protein pocket for downstream tasks such as protein-ligand docking that will specify more precise protein-ligand interface around the predicted protein pocket.** We have studied P2Rank[4] in our paper (Section A3.1). 

Most current deep learning methods [6, 7, 8] for protein pocket prediction use CNNs and require voxelization of protein structures. 
EquiPocket innovatively uses a geometric-aware E(3)-equivariant GNN, which is a unique model unexplored by other research in this field. 
As also noted by Reviewer 9vNs, "the idea of using an E(3)-equivariant GNN for binding site prediction is of interest especially in the bioinformatic domain."

Reviewer comment (Experiments):
I think it might be also necessary to compare with protein-ligand docking methods as binding site prediction is one of their outputs.

Author response:
Thank your for the comment. Again, we highlight that protein-ligand docking and protein pocket prediction are two different tasks. A general pipeline in current methods (E3Bind [1] and TankBind [2]) is that we first predict the protein pocket, and then conduct protein-ligand docking and infer how specific ligands interact with the atoms around the predicted pocket. Although the protein-ligand docking methods also output binding site, this area is a fine-grained subset of the atoms around the protein pocket. Hence, it is not reasonable to  compare with protein-ligand docking methods. We have further introduced the difference between these two tasks in the revision.

Reviewer comment (Evaluation):
It would be better to also compare the inference speed of different methods. Because this approach does massage passing on full protein atoms graph, it appears the inference speed would be very slow.

Author response:
Thank you for the feedback. The comparison of various methods for predicting 100 proteins reveals the following:
 
|   Method   |       Type      | Time (s)  per 100 proteins | Average DCC |
|:----------:|:---------------:|:--------------------------:|:-----------:|
|   fpocket  | Geometric-based |             23             |    0.214    |
|  Kalasanty | 3D-CNN          |             86             |    0.321    |
|  DeepSurf  | 3D-CNN          |             641            |    0.366    |
| EquiPocket |       Ours      |             37             |    0.431    |


| Dataset  | Average  Atom Num | Average  Atom in Surface | Average the True Center of Binding Sites  |
|----------|-------------------|--------------------------|-------------------------------------------|
| COACH420 | 2123              | 1217                     | 1.2                                       |
| HOLO4K   | 3845              | 2052                     | 2.4                                       |
| PDBbind  | 3104              | 1677                     | 1                                         |



fpocket[5]: Fastest with only 23 seconds for 100 proteins, leveraging manually defined geometric features. However, its performance metrics are not notable.

Kalasanty[6] and DeepSurf[7]: Both are 3D-CNN-based. DeepSurf, using detailed local grids on protein surfaces, outperforms Kalasanty in metrics but is slower and the least efficient among the methods compared.

EquiPocket: Our method takes 47 seconds per 100 proteins and shows the best DCC metrics. It's faster than 3D-CNN methods but slower than geometric-based ones. This is due to 3D-CNN methods transforming proteins into 3D images (eg. 36 * 36 * 36 grids~[6, 7, 8]), increasing computational costs compared to our using atom information (average 2000-3000 nodes in a protein). EquiPocket also integrates surface features with Surface-EGNN, enhancing efficiency over DeepSurf.

We have included these results and analyses in the revised paper. Thank you for your nice suggestion.

Reviewer comment (Experiments):
The setting for some baseline methods (EGNN, SchNet) needs to be more clear. For example, what kinds of graphs do you use for the EGNN? Do you also use the surface atom graph?

Author response:
We focused on using EGNN[10] and SchNet[11] solely on the protein structure graph for two main reasons: 

Both EGNN and SchNet were originally developed for representing molecular nodes and structures, making them ideal to test whether solely taking protein structural information is sufficient for predicting ligand binding sites. 

Our method EquiPocket differs significantly from previous prediction method [4, 5, 6, 7, 8] by employing surface atom graph, which represents an innovative contribution and differentiates from the conventional protein structure graph as baseline models.

Reviewer comment (Experiments):
A more informative ablation would involve comparing EquiPocket, EquiPocket/L, EquiPocket/R, and EquiPocket/LR, where the symbol '/' means exclude. This is because L and R appear to be two feature extractors.

Author response:
Our model predominantly comprises two feature extractors: local geometric modeling module and global structural modeling module, subsequently followed by the surface-EGNN model. In response to your valuable suggestion, we propose the following definitions: 

EquiPocket/L: This variant of EquiPocket excludes local geometric modeling module. 

EquiPocket/R: This variant of EquiPocket excludes global structural modeling module. 

EquiPocket/LR: This variant of EquiPocket excludes both local geometric modeling module and global structural modeling module. 

| Dataset       |            | COACH420 |        | HOLO4k |        | PDBbind |        |
|---------------|------------|----------|--------|--------|--------|---------|--------|
| Model         | Fail Ratio | DCC      | DCA    | DCC    | DCA    | DCC     | DCA    |
| EquiPocket/L  | 0.13       | 0.355    | 0.546  | 0.296  | 0.574  | 0.465   | 0.606  |
| EquiPocket/R  | 0.09       | 0.364    | 0.541  | 0.294  | 0.598  | 0.474   | 0.627  |
| EquiPocket/LR | 0.16       | 0.308    | 0.502  | 0.268  | 0.543  | 0.409   | 0.566  |
| EquiPocket    | 0.05       | 0.423    | 0.656  | 0.337  | 0.662  | 0.545   | 0.721  |

Analysis shows that omitting any of these modules negatively impacts performance. Specifically, excluding either the local geometric (L) or global structural (R) module leads to a 10\%-15\% decrease in DCC/DCA metrics; removing both L and R modules results in a more significant drop of 20\%-25\%. 
These results highlight the essential role of both feature extractors in predicting ligand binding sites. Notably, the more pronounced performance decline when omitting the local geometric module (L) suggests its higher importance in protein pocket prediction. This finding is consistent with current trends where methods like Fpocket[5], P2rank[4], and DeepSurf[7] primarily utilize geometric features for binding site prediction.

Reviewer comment (Experiments):
The authors did not compare their method with latest state-of-the-art methods, such as (1). (1) Tubiana J, Schneidman-Duhovny D, Wolfson H J. ScanNet: an interpretable geometric deep learning model for structure-based protein binding site prediction[J]. Nature Methods, 2022, 19(6): 730-739.

Author response:
For the ScanNet[9] method, we indeed explored this method during our research's early stages, as detailed in section A.3.3 of our paper. We attempted to compare EquiPocket with ScanNet but eventually opted against it due to the specific reasons:

**Task Discrepancy.** EquiPocket is focused on predicting ligand binding sites, whereas ScanNet is designed for predicting binding sites in protein and protein, protein and antibody, and protein and disordered protein interactions, making it unsuitable for our specific task.

**Incompatibility in Application.** ScanNet's pretrained model, dataset, and web service are tailored for its specific domains (details in provided [figures](https://z1.ax1x.com/2023/11/14/piYeQfS.png) and their [webserver](http://bioinfo3d.cs.tau.ac.il/ScanNet/index_real.html)). When applied to ligand binding site prediction, ScanNet yielded inconsistent results with the test datasets.

Reviewer comment (Experiments):
Some details need to be clarified. For example, proteins have multiple binding sites. How did the authors select binding sites.

Author response:
Thank you for your valuable guidance. The below table categorizes samples in test datasets based on the number of ligands binding to a protein. The majority of samples in the test dataset consist of a single binding site, with only a subset of samples containing multiple binding sites. 

| The number of ligands binding to a protein | COACH420 | Holo4k | PDBbind |
|:----------------------------------------:|:--------:|:------:|:-------:|
|                     1                    |    235   |  2442  |   5025  |
|                     2                    |    36    |   635  |    0    |
|                     3                    |     7    |   67   |    0    |
|                     4                    |     4    |   22   |    0    |
|                    >=5                   |     2    |   38   |    0    |


Following related methods [4, 5, 6, 7, 8], the specific processing steps of our method are outlined below, and further details can be found in Section 4.1 and Appendix A.1 in our paper: 

**Training Process**: 

a. Following DeepSurf [7], the protein atoms within 4 Ångströms of any ligand atom are set as positive, otherwise negative. 

b. EquiPocket predicts the druggability probability for each atom, using the Dice loss function for model optimization. 

**Validation and Testing**: 

a. In accordance with related methods [4, 5, 6, 7, 8], we define the center of a binding site as the mean position of the ligand atoms. This definition allows us to handle the proteins with multiple binding sites (N >= 1). 

b. We focus on predicting druggability probabilities for each atom rather than identifying specific binding site atoms. 

c. Our method uses predicted atom-level probabilities in a clustering process [10] to autonomously identify ligand binding site centers and druggability, which are crucial for metrics and downstream docking tasks.  

**Performance Evaluation**: 

 a. We evaluate based on the top-$n$ predicted binding site centers, where $n$ does not exceed the actual number of binding sites. If no predicted binding sites center is found, it is considered a failure. 

 b. The predicted ligand binding center with the DCC/DCA falls below a predetermined threshold, typically set to 4, is classified as successful. 

Through the aforementioned process, we are able to effectively handle proteins with more than one binding site during training, validation, and testing.

Reviewer comment (Theory):
There is a significant correlation between binding sites and the properties of small molecules, not just spatial relationships.

Author response:
We fully understand your concerns. Indeed, a single binding site might bind to multiple ligands, with the binding interface varying based on each ligand's structure and properties. However, these variations minimally impact our main task. Ligand binding sites on a protein are generally seen as fixed regions, defined as a fixed-size box around a designated center [1, 2, 3, 4, 5, 6, 7, 8, 9]. This remains constant despite slight variations in how different ligands interact with the same binding site. More importantly, in our dataset as showed in table of Q2, most proteins have only one binding site, and usually, each binding site interacts with a single ligand. We have included the abve explanations in the revised paper to address your concern.

Reviewer comment (Novelty):
The significance of the studied problem is limited, given lots of ligand docking prediction models and structure-based drug design models are proposed. It would be better if the authors could show how much improvement the proposed method can bring to the downstream tasks.

Author response:
Thank you for the comment. As already demonstrated in previous studies [4, 5, 6, 7, 8], ligand binding site (or pocket) prediction comes as a fundamental step for many downstream tasks. For example, the models such as E3Bind [1], TankBind [2], and DeepDock [3] mainly focus on the tasks like protein-ligand binding pose and affinity prediction. They often use tools like P2rank [4] to identify target pocket for later fine-grained 
prediction of ligand docking pose. Our EquiPocket shares the same function as P2rank and can be applied in broad docking methods. It will be interesting to see the actual performance improvement in downstream tasks, which, however, requires end2end experimental evaluations and could be better left for future exploration.

Reviewer comment (Experiments):
Why is relative direction prediction needed? The ablation study about this loss is needed.

Author response:
The core reasons are as follows: Firstly, for ligand binding sites prediction task[4, 5, 6, 7, 8], the ligand cannot be used as input data and can only be employed to define the target. Using it as input would risk data leakage. Secondly, the relative direction between a protein atom and its nearest ligand atom effectively captures the local geometric information of the binding sites on a protein. Different from the previous works [6, 7, 8], our method can output E(3)-equivariant coordinates (detailed in Section 3.3 of our paper). To better capture the geometric details of the protein surface, we introduced more detailed relative direction as a supplementary target. 

The corresponding ablation results have been shown in Figure 6 (c) of our original paper, and detailed results are presented below. 
EquiPocket (w/o Direction Loss) represents our EquiPocket method removing the relative direction prediction module. 
It can be observed that when the relative direction prediction module is removed, our method's performance drops for proteins of different sizes. This is especially notable for proteins with fewer than 3000 atoms, which account for 60% of the samples. If the relative direction prediction is removed, the performance drops by approximately 10%. 
These results demonstrate the effectiveness of our designed relative direction target. 

| Atom Num  | Protein Num | Ratio | Cumsum Ratio | DCC of EquiPocket(w/o Direction loss) | DCC of EquiPocket |
|-----------|-------------|-------|--------------|--------------------------------|------------|
| 0-1000    | 296         | 0.04  | 0.04         | 0.319                          | 0.428      |
| 1000-2000 | 2193        | 0.27  | 0.3          | 0.587                          | 0.621      |
| 2000-3000 | 2534        | 0.31  | 0.61         | 0.551                          | 0.590       |
| 3000-4000 | 1143        | 0.14  | 0.75         | 0.371                          | 0.388      |
| 4000-5000 | 619         | 0.08  | 0.82         | 0.258                          | 0.255      |
| >=5000    | 1440        | 0.18  | 1            | 0.144                          | 0.153      |

Reviewer comment (Theory):
My main concern is that the proposed method is unaware of the structure of ligand molecules (or proteins). In a real-world setting, the binding site of a target protein might be dependent on the ligands they bind to.

Author response:
We fully understand your concerns, and there may be some misunderstanding in this context. 

The task of our EquiPocket is ligand binding site prediction. **The key output of our method is identifying the center of candidate binding sites on the protein, rather than detailing the exact protein-ligand interface**.  The area around the center is seen as candidate binding sites for further docking tasks, and it is unnecessarily related to specific ligands. For example, the docking methods such as E3Bind [1] and TankBind [2] first use existing methods to predict the binding site of the protein, and then conduct docking process by considering the interaction between the ligand and the protein  around the binding site.

The below table categorizes samples in test datasets based on the number of ligands binding to a protein. 
Indeed, most proteins have only one ligand binding site, though some can bind with multiple ligands. The binding interfaces depend on the structure and properties of each ligand. However, these variations minimally impact the task. Ligand binding sites on a protein are generally seen as fixed regions, defined as a fixed-size box around a designated center[1, 2, 3, 4, 5, 6, 7, 8, 9]. This remains constant despite slight variations when different ligands interact with the same binding site.

| The number of ligands binding to a protein | COACH420 | HOLO4k | PDBbind |
|:----------------------------------------:|:--------:|:------:|:-------:|
|                     1                    |    235   |  2442  |   5025  |
|                     2                    |    36    |   635  |    0    |
|                     3                    |     7    |   67   |    0    |
|                     4                    |     4    |   22   |    0    |
|                    >=5                   |     2    |   38   |    0    |

Reviewer comment (Experiments):
P2rank is a widely used package in literature (e.g., TANKBIND[1], E3BIND[2]) for locating the ligand-binding pockets based on protein structures. It is quite accurate and efficient. The authors should include it as a baseline method.

Author response:
Thank you for your suggestions. We have studied P2Rank [4] in our paper (Section A3.1). However, we didn't include P2Rank as a baseline mainly due to **the differences in training and validation data between it and deep learning methods including Kalasanty [6], DeepSurf [7], deepsite [8], and our EquiPocket.** Specifically, P2Rank uses data from CHEN11 and JOINED datasets, while deep learning methods commonly use scPDB[10].P2Rank's paper mentions (details in [here](https://z1.ax1x.com/2023/11/15/piYcPNn.png)) that CHEN11 is more diverse than scPDB , affecting model performance. The comparison results are as following: 

| DCA                | COACH420 | HOLO4K |
|--------------------|----------|--------|
| P2Rank[protrusion] | 0.642    | 0.593  |
| P2rank             | 0.683    | 0.706  |
| DeepSite           | 0.564    | 0.456  |
| Kalasanty          | 0.636    | 0.515  |
| deepsurf           | 0.658    | 0.635  |
| EquiPocket         | 0.656    | 0.662  |

The results in above table show that our method essentially matches or surpasses most deep learning methods, even outperforming P2Rank [protrusion], which uses only geometric information, and slightly trailing behind P2Rank, which benefits from a more diverse dataset.  
**These results has been added to the revised paper.**

Reviewer comment (Theory):
What is the rationale behind Eq.3? Why do you swap the order or MLP and Pooling?

Author response:
Eq.3: $g_{i}=[Pooling( \\{ MLP(g(s_{j})) \\} \_{s_{j} \in S_{i}} ), MLP(Pooling(\\{g(s_{j})\\}\_{s_{j} \in S_{i}}))]$

In Eq.3, $g_i$ is the geometric embedding for a protein atom, learned from surrounding surface probes, and $s_i$ denotes the local geometric properties of these probes with properties such as distances and angle to protein atoms, the surface center, neighboring probes, and so on. 
Initially, we apply an MLP to these features, followed by pooling. This process transforms the geometric properties before aggregating them into the protein node's geometric embedding. 
However, since each property of  $s_i$ itself carries meaningful information. We are concerned that applying MLP first and then pooling might weaken the transmission of this information. Therefore, we take the second part of the equation. 

|                   |            | Coach420 |        | HOLO4k |        | PDBbind |        |
|:-----------------:|:----------:|:--------:|:------:|:------:|:------:|:-------:|:------:|
|       Model       | Fail Ratio |    DCC   |   DCA  |   DCC  |   DCA  |   DCC   |   DCA  |
| EquiPocket-former |    0.16    |  0.389   | 0.606  | 0.330  | 0.637  |  0.507  | 0.660  |
| EquiPocket-latter |    0.16    |  0.407   | 0.617  | 0.319  | 0.644  |  0.529  | 0.676  |
|     EquiPocket    |    0.05    |  0.423   | 0.656  | 0.337  | 0.662  |  0.545  | 0.721  |

To highlight the effectiveness of the features in Eq.3, we carried out extra experiments with "EquiPocket-former" focusing on the equation's initial part and "EquiPocket-latter" on its latter part. The findings show: 
Using either feature alone diminishes the predictive performance compared to the full EquiPocket model. Specifically, "EquiPocket-former" alone sees about a 10% drop, while "EquiPocket-latter" alone results in around a 5% reduction. 
This outcome underscores the necessity of both features, with the latter part having a more substantial impact on our model’s performance.

Reviewer comment (Experiments):
Can the proposed method be adapted to protein-protein binding site prediction?

Author response:
Nice suggestion. However, in our current version, we focus only on ligand-protein binding site prediction for these reasons:

a. Our method is specifically trained on a dataset for ligand binding sites [10]. Therefore, directly applying it for protein-protein binding site prediction is not feasible due to the specialized nature of the training.

b, While predicting protein-ligand and protein-protein binding sites may have some similarities, the differences in the types of interacting partners (ligands vs. proteins), their sizes, properties, and their effects on the target protein are significant. These differences likely require distinct model and feature design. 

We respectfully understand the reviewer's suggestion and are willing to extend the application scope of our method in future exploration.