Uni-Mol: A Universal 3D Molecular Representation Learning Framework
Abstract
Molecular representation learning (MRL) has gained tremendous attention dueto its critical role in learning from limited supervised data for applications likedrug design.
In most MRL methods, molecules are treated as 1D sequential tokensor 2D topology graphs, limiting their ability to incorporate 3D information fordownstream tasks and, in particular, making it almost impossible for 3D geometryprediction or generation.
Herein, we propose Uni-Mol, a universal MRL frameworkthat significantly enlarges the representation ability and application scope of MRLschemes.
Uni-Mol is composed of two models with the same SE(3)-equivarianttransformer architecture: a molecular pretraining model trained by 209M molecularconformations; a pocket pretraining model trained by 3M candidate protein pocketdata.
The two models are used independently for separate tasks, and are combinedwhen used in protein-ligand binding tasks.
By properly incorporating 3D information, Uni-Mol outperforms SOTA in 14/15 molecular property prediction tasks.
Moreover, Uni-Mol achieves superior performance in 3D spatial tasks, includingprotein-ligand binding pose prediction, molecular conformation generation, etc.
Finally, we show that Uni-Mol can be successfully applied to the tasks withfew-shot data like pocket druggability prediction.
The model and data will bemade publicly available at https://github.com/dptech-corp/Uni-Mol .
Introduction
Recently, representation learning (or pretraining, self-supervised learning)
[1, 2, 3] has been prevailingin many applications, such as BERT [4] and GPT [5, 6, 7] in Natural Language Processing (NLP),
ViT [8] in Computer Vision (CV), etc.
These applications have a common characteristic: unlabeleddata is abundant, while labeled data is limited.
As a solution, in a typical representation learningmethod, one first adopts a pretraining procedure to learn a good representation from large-scaleunlabeled data, and then a finetuning scheme is followed to extract more information from limitedsupervised data.
Applications in the field of drug design share the characteristic that calls for representation learningschemes.
The chemical space that a drug candidate lies in is vast, while drug-related labeled data islimited.
Not surprisingly, compared with traditional molecular fingerprint based models [9, 10], recentmolecular representation learning (MRL) models perform much better in most property predictiontasks [11, 12, 13].
However, to further improve the performance and extend the application scopeof existing MRL models, one is faced with a critical issue.
From the perspective of life science, theproperties of molecules and the effects of drugs are mostly determined by their 3D structures [14,15].
In most current MRL methods, one starts with representing molecules as 1D sequential strings,such as SMILES [16, 17, 18] and InChI [19, 20, 21], or 2D graphs [22, 11, 23, 12, 24].
This maylimit their ability to incorporate 3D information for downstream tasks.
In particular, this makes italmost impossible for 3D geometry prediction or generation, such as, e.g., the prediction of protein-
[Figure]
ligand binding pose [25].
Even though there have been some recent attempts trying to leverage 3Dinformation in MRL [26, 27], the performance is less than optimal, possibly due to the small size of3D datasets, and 3D positions can not be used as inputs/outputs during finetuning, since they onlyserve as auxiliary information.
In this work, we propose Uni-Mol, to our best knowledge, the first universal 3D molecular pretrainingframework, which is derived from large-scale unlabeled data and is able to directly take 3D positionsas both inputs and outputs.
Uni-Mol consists of 3 parts.
1) Backbone .
Based on Transformer, theinvariant spatial positional encoding and pair level representation are added to better capture the 3Dinformation.
Moreover, an equivariant head is used to directly predict 3D positions.
2) Pretraining .
We create two large-scale datasets, a 209M molecular conformation dataset and a 3M candidateprotein pocket dataset, for pretraining 2 models on molecules and protein pockets, respectively.
For the pretraining tasks, besides masked atom prediction, a 3D position denoising task is usedfor learning 3D spatial representation.
Finetuning .
According to specific downstream tasks, theused pretraining models are different.
For example, in molecular property prediction tasks, only themolecular pretraining model is used; in protein-ligand binding pose prediction, both two pretrainingmodels are used.
We refer to Fig.
1 for an overall schematic illustration of the Uni-Mol framework.
To demonstrate the effectiveness of Uni-Mol, we conduct experiments on a series of downstreamtasks.
In the molecular property prediction tasks, Uni-Mol outperforms SOTA on 14/15 datasets onthe MoleculeNet benchmark.
In 3D geometric tasks, Uni-Mol also achieves superior performance.
For the pose prediction of protein-ligand complexes, Uni-Mol predicts 88.07% binding poses with RMSD < = 2Å, 22.81% more than popular docking methods, and ranks 1st in the docking power teston CASF-2016 [28] benchmark.
Regarding molecular conformation generation, Uni-Mol achieves SOTA for both Coverage and Matching metrics on GEOM-QM9 and GEOM-Drugs [29].
Moreover, Uni-Mol can be successfully applied to tasks with very limited data like pocket druggability prediction.
Uni-Mol Framework
In this section, we introduce the Uni-Mol framework by showing the details of the backbone, thepretraining scheme, and the finetuning scheme.
We refer to Fig.
2 for a schematic illustration of themodel architecture.
[Figure]
𝑚𝑜𝑙𝑒𝑐𝑢𝑙𝑒 𝑝𝑜𝑐𝑘𝑒𝑡
2.1 Backbone
[Figure] Transformer [30] is widely used as a backbone model in representation learning.
However, Transformer was originally designed for NLP tasks and cannot handle 3D spatial data directly.
To tacklethis, based on the standard Transformer with Pre-LayerNorm [31] backbone, we introduce severalmodifications.
Invariant spatial positional encoding Due to its permutationally invariant property, Transformercannot distinguish the positions of inputs without positional encoding.
Different with the discrete(ordinal) positions used in NLP/CV [32, 33], the positions in 3D space, i.e. coordinates, are continuousvalues.
Besides, the positional encoding procedure needs to be invariant under global rotation andtranslation.
To achieve that, similar to the relative positional encoding, we simply use Euclideandistances of all atom pairs, as well as pair-type aware Gaussian kernels [34].
Formally, the D -channelpositional encoding of atom pair ij is denoted as
[Equation]
where G ( d, µ, σ ) =
[Equation]
is a Gaussian density function with parameters µ and σ , d ij is the Euclidean distance of atom pair ij , and t ij is the pair-type of atom pair ij .
Please note the pair-typehere is not the chemical bond, and it is determined by the atom types of pair ij .
A ( d ij , t ij ; a , b ) isthe affine transformation with parameters a and b , it affines d ij corresponding to its pair-type t ij .
Except d ij and t ij , all remaining parameters are trainable and randomly initialized.
Pair representation By default, Transformer maintains the token(atom) level representation, whichis later used in finetuning downstream tasks.
Nevertheless, as the spatial positions are encoded atpair-level, we also maintain the pair-level representation, to better learn the 3D spatial representation.
Specifically, the pair representation is initialized as the aforementioned spatial positional encoding.
Then, to update pair representation, we use the atom-to-pair communication via the multi-head QueryKey product results in self-attention.
Formally, the update of ij pair representation is denoted as
[Equation]
where q lij is the pair representation of atom pair ij in l -th layer, H is the number of attention heads,d is the dimension of hidden representations, Q l,hi ( K l,hj ) is the Query (Key) of the i -th ( j -th) atomin the l -th layer h -th head, and
M ∈ R D
× H is the projection matrix to make the representation thesame shape as multi-head Query-Key product results.
Besides, to leverage 3D information in the atom representation, we also introduce the pair-to-atomcommunication, by using the pair representation as the bias term in self-attention.
Formally, the self-attention with pair-to-atom communication is denoted as
[Equation]
where V l,hj is the Value of the j -th atom in the l -th layer h -th head.
The pair representation andatom-pair communication are firstly proposed in the Evoformer in AlphaFold [35], but the cost of
Evoformer is extremely large.
In Uni-Mol, as we keep them as simple as possible, the extra cost ofmaintaining pair representation is negligible.
SE(3)-Equivariance coordinate head With 3D spatial positional encoding and pair representation,the model can learn a good 3D representation.
However, it still lacks the ability to directly output coordinates, which is essential in 3D spatial tasks.
To this end, we add a simple SE(3)-equivariance headto Uni-Mol.
Following the idea of EGNN [36], the design of SE(3)-equivariance head is denoted as
[Equation]
where n is the number of total atoms, L is the number of layers in model, x i ∈ R 3 is the inputcoordinate of i -th atom, and ˆ x i ∈ R 3 is the output coordinate of i -th atom, ReLU ( y )
= max (0 , y )is Rectified Linear Unit [37],
U ∈ R H
× H and W ∈ R H × 1 are the projection matrices to convertpair representation to scalar.
2.2 Pretraining
For the purpose of pretraining, we generate two large-scale datasets, one composed of 3D structuresof organic molecules, and another composed of 3D structures of candidate protein pockets.
Then,two models are pretrained using these two datasets, respectively.
As pockets are directly involvedin many drug design tasks, intuitively, the pretraining on candidate protein pockets can boost theperformance of tasks related to protein-ligand structures and interactions.
The molecular pretraining dataset is based on multiple public datasets (See Appendix ?? for moreinformation).
After normalizing and deduplicating, it contains about 19M molecules.
To generate3D conformations, we use ETKGD [38] with Merck Molecular Force Field [39] optimizationin RDKit [40] to randomly generate 10 conformations for each molecule.
We also generate anadditional 2D conformation (based on the molecular graph), to avoid some rare cases that fail togenerate 3D conformations.
The protein pocket pretraining dataset is derived from the Protein Data Bank (RCSB PDB 1 ) [41], acollection of 180K 3D structures of proteins.
To extract candidate pockets, we first clean the databy adding the missing side chains and hydrogen atoms; then we use Fpocket [42] to detect possiblebinding pockets of the proteins; and finally, we filter pockets by the number of residues in contactwith and retains water molecules in the pocket.
In this way, We collect a dataset composed of 3.2Mcandidate pockets for pretraining.
Self-supervised task is vitally important for effective learning from large-scale unlabeled data.
For example, the masked token prediction task in BERT [4] encourages the model to learn thecontextual information.
Similar to BERT, the masked atom prediction task is used in Uni-Mol.
For each molecule/pocket, we add a special atom [CLS] , whose coordinate is the center of allatoms, to represent the whole molecule/pocket.
However, as 3D spatial positional encoding leakschemical bonds, atom types could be inferred easily, and therefore, the masked atom predictioncannot encourage the model to learn useful information.
To tackle this, as well as learning from 3Dinformation, we design a 3D position denoising task.
Particularly, uniform noises of areadded to the random 15% atom coordinates, then the spatial positional encoding is calculated basedon corrupted coordinates.
In this way, the masked atom prediction task becomes non-trivial.
Besides,two additional heads are used to recover the correct spatial positions.
1) Pair-distance prediction.
Based on pair-representation, the model needs to predict the correct Euclidean distances of the atomspairs with corrupted coordinates.
2) Coordinate prediction.
Based on SE(3)-Equivariance coordinatehead, the model needs to predict the correct coordinates for the atoms with corrupted coordinates.
Both 2 pretraining models use the same self-supervised tasks described above, and Figure 2 is theillustration of the overall pretraining framework.
For the detailed configurations of pretraining, pleaserefer to Appendix ?? .
2.3 Finetuning
To be consistent with pretraining, we use the same data prepossessing pipeline during finetuning.
For molecules, as multiple random conformations can be generated in a short time, we can use themas data augmentation in finetuning to improve performance and robustness.
Some molecules may failto generate 3D conformations, and we use their molecular graph as 2D conformation.
For tasks thatprovide atom coordinates, we use them directly and skip the 3D conformation generation process.
As there are 2 pretraining models and several types of downstream tasks, we should properly usethem in the finetuning stage.
According to the task types, and the involvement of protein or ligand,we can categorize them as follow.
Non-3D prediction tasks These tasks do not need to output 3D conformations.
Examples includemolecular property prediction, molecule similarity, pocket druggability prediction, protein-ligandbinding affinity prediction, etc.
Similar to NLP/CV, we can simply use the representation of [CLS]which represents the whole molecule/pocket, or the mean representation of all atoms, with a linearhead to finetune on downstream tasks.
In the tasks with pocket-molecule pair, we can concatenatetheir [CLS] representations, and then finetune with linear head.3D prediction tasks of molecules or pockets These tasks need to predict a 3D conformationof the input, such as molecular conformation generation.
Different with the fast conformationgeneration method used in Uni-Mol, molecular conformation generation task usually requires runningadvanced sampling and semi-empirical density functional theory (DFT) to account for the ensembleof 3D conformers that are accessible to a molecule, and this is very time-consuming.
Therefore,there are many recent works that train the model to fast generate conformations from moleculargraph [43, 44, 45, 46].
While in Uni-Mol, this task straightforwardly becomes a conformationoptimization task: generate a new conformation based on a different input conformation.
Specifically,in finetuning, the model supervised learns the mapping from Uni-Mol generated conformations tothe labeled conformations.
Moreover, the optimized conformations can be generated end-to-end by
SE(3)-Equivariance coordinate head.3D prediction tasks of protein-ligand pairs This is one of the most important tasks in structurebased drug design.
The task is to predict the complex structure of a protein binding site and amolecular ligand.
Besides the conformation changes of the pocket and the molecule themselves, wealso need to consider how the molecule lays in the pocket, that is, the additional 6 degrees (3 rotationsand 3 translations) of freedom of a rigid movement.
In principle, with Uni-Mol, we can predict thecomplex conformation by the SE(3)-Equivariant coordinate head in an end-to-end fashion.
However,this is unstable as it is very sensitive to the initial docking positions of molecular ligand.
Herein, toget rid of the initial positions, we use a scoring function based optimization method in this paper.
Inparticular, the molecular representation and pocket representation are firstly obtained from their ownpretraining models by their own conformations; then, their representations are concatenated as theinput of an additional 4-layer Uni-Mol decoder, which is finetuned to learn the pair distances of allatoms in molecule and pocket.
With the predicted pair-distance matrix as the scoring function, weuse a simple differential evolution algorithm [47] to sample and optimize the complex conformations.
More details can be found in Appendix ?? .
Experiments
To verify the effectiveness of our proposed Uni-Mol model, we conduct extensive experimentson
multiple downstream tasks, including molecular property prediction, molecular conformationgeneration, pocket property prediction, and protein-ligand binding pose prediction.
Besides, we alsoconduct several ablation studies.
Due to space restrictions, we leave the detailed experimental settingsand ablation studies to Appendix ?? .
3.1 Molecular property prediction
Datasets and setup MoleculeNet [48] is a widely used benchmark for molecular propertyprediction, including datasets focusing on different levels of properties of molecules, from quantum
[Table] [Table]
[Table] [Table]
- - - MolCLR
[Table]
mechanics and physical chemistry to biophysics and physiology.
Following previous work GEM [13],we use scaffold splitting for the dataset and report the mean and standard deviation of the resultsfor three random seeds.
Baselines We compare Uni-Mol with multiple baselines, including supervised and pretrainingbaselines.
D-MPNN [49] and AttentiveFP [50] are supervised GNNs methods.
N-gram [51],
PretrainGNN [22], GROVER [11], GraphMVP [26], MolCLR [12], and GEM [13] are pretrainingmethods.
N-gram embeds the nodes in the graph and assembles them in short walks as the graphrepresentation.
Random Forest and XGBoost [52] are used as the predictor for downstream tasks.
Results Table 1 and Table 2 show the experiment results of Uni-Mol and competitive baselines,where the best results are marked in bold.
Most baseline results are from the paper of GEM, except forthe recent works GraphMVP and MolCLR.
The results of GraphMVP are from its paper.
As MolCLRuses a different data split setting (without considering chirality), we rerun it with the same data splitsetting as other baselines.
From the results, we can summarize them as follows: 1) overall, Uni-Moloutperforms baselines on almost all downstream datasets.
In solubility (ESOL, Lipo), free energy(FreeSolv), and quantum mechanical (QM7, QM8, QM9) properties prediction tasks, Uni-Mol issignificantly better than baselines.
As 3D information is critical in these properties, it indicates that
Uni-Mol can learn a better 3D representation than other baselines.
3) Uni-Mol fails to beat SOTA onthe SIDER dataset.
After investigation, we find Uni-Mol fails to generate 3D conformations (androllbacks to 2D graphs) for many molecules (like natural products and peptides) in SIDER.
Therefore,due to the missing 3D information, it is reasonable that Uni-Mol cannot outperform others.
In summary, by better utilizing 3D information in pretraining, Uni-Mol outperforms all previous
MRL models in almost all property prediction tasks.
[Table]
3.2 Molecular conformation
generation
Datasets and setup Following the settings in previous works [44, 53], we use GEOM-QM9 and
GEOM-Drugs [54] dataset to perform conformation generation experiments.
As described in Sec.
2.3,in this task, Uni-Mol optimizes its generative conformations to the labeled ones.
To construct thefinetuning data, we first randomly generate 10 conformations.
Then, for each of them, we calculatethe RMSD between it and labeled conformations, and choose the one with minimal RMSD as itsoptimizing target.
For the inference in the test set, we generate the same number of conformations(twice the number of labeled conformations) as previous works do.
And we use the same metrics,
Coverage (COV) and Matching (MAT).
Higher COV means better diversity, while lower MAT meanshigher accuracy.
Baselines We compare Uni-Mol with 10 competitive baselines.
RDKit [38] is a traditional conformation generation method based on distance geometry.
The rest baseline can be categorized into twoclasses.
GraphDG [43], CGCF[44], ConfVAE [55], ConfGF [53], and DGSM [56] combine generative models with distance geometry, which first generates interatomic distance matrices and theniteratively generates atomic coordinates.
CVGAE [45], GeoMol [46], DMCG [57], and GeoDiff [58]directly generate atomic coordinates.
Results The results are shown in Table 3.
We report the mean and median of COV and MAT on
GEOM-QM9 and GEOM-Drugs datasets.
ConfVAE [55], GeoMol[46], DGSM [56], DMCG [57],
GeoDiff’s [58] results are from their papers, respectively.
Other baseline results are from ConfGF’spaper.
As shown in Table 3, Uni-Mol exceeds existing baselines in both COV and MAT metrics onboth datasets.
Although Uni-Mol outperforms SOTA, we suspect that the above benchmark cannotsatisfy the real-world demand of conformation generation tasks in the field of drug design.
Sincethe ensemble of molecular conformations in biological systems is different from that in a vacuum orgeneral solution environment, the ensemble of bioactive conformation must be considered in order toapply the conformation generation model in the context of drug design, while the GEOM dataset justignores this.
Establishing a reasonable benchmark will be crucial in this research direction.
3.3 Pocket property prediction
Datasets and setup Druggability, the ability of a candidate protein pocket to produce stablebinding to a specific molecular ligand, is one of the most critical properties of a candidate proteinpocket.
However, this task is very challenging due to the very limited supervised data.
For example,
NRDLD [59], a commonly used dataset, only contains 113 data samples.
Therefore, besides
NRDLD, we construct a regression dataset for benchmarking pocket property prediction performance.
Specifically, based on Fpocket tool, we calculate Fpocket Score, Druggability Score, Total SASA,and Hydrophobicity Score for the selected 164,586 candidate pockets.
Model is trained to predictthese scores.
To avoid leaking, the selected pockets are not overlapped with the candidate proteinpocket dataset used in Uni-Mol pretraining.
Baselines On the NRDLD dataset, we compare Uni-Mol with 6 previous methods evaluated in [60].
Accuracy, recall, precision, and F1-score are used as metrics for this classification task.
On ourcreated benchmark dataset, as there are no appropriate baselines, we use an additional Uni-Mol model
[Table]
without pretraining, denoted as Uni-Mol random , to check the performance brought by pretraining onpocket property prediction.
MSE (mean square error) is used as the metric.
Results As shown in Table 4, Uni-Mol shows the best accuracy, recall, and F1-score on NRDLD,the few-show dataset.
In our created benchmark dataset, the pretraining Uni-Mol model largelyoutperforms the non-pretraining one on all four scores.
This indicates that pretraining on candidateprotein pockets indeed brings improvement in pocket property prediction tasks.
Unlike Molecular property prediction, due to the very limited supervised data, pocket propertyprediction gained much less attention.
Therefore, we also plan to release our created benchmarkdataset, and hopefully, it can help future research.
3.4 Protein-ligand binding pose prediction
Datasets and setup As mentioned above, protein-ligand binding pose prediction is one of the mostimportant tasks in drug design.
And Uni-Mol combines both the molecular and pocket pretrainingmodels to learn a distance matrix based scoring function, and then sample and optimize the complexconformations.
For the benchmark dataset, referring to the previous works [28, 61], we use CASF2016 as the test set.
For the training data used in finetuning, we use PDBbind General set v.2020 [62](19,443 protein-ligand complexes), excluding complexes that already exist in the CASF-2016.
Two benchmarks are conducted: 1) Docking power, the default metric to benchmark the ability of ascoring function in CASF-2016.
Specifically, it tests whether a scoring function can distinguish theground truth binding pose from a set of decoys or not.
For each ground truth, CASF-2016 provides50 100 decoy conformations of the same ligand.
Scoring functions are applied to rank them, and theground truth binding pose is expected to be the top 1. 2)
Binding pose accuracy.
Specifically, we usethe semi-flexible docking setting: keep the pocket conformation fixed, while the conformation of theligand is fully flexible.
We evaluate the RMSD between the predicted binding pose and the groundtruth.
Following previous works, we use the percentage of results that are below predefined RMSDthresholds as metrics.
Baselines For docking power benchmark, the baselines are DeepDock [61] and the top 10 scoringfunctions reported in [28], including both conventional scoring functions and machine learningbased ones.
For the binding pose accuracy, the baselines are Autodock Vina [63, 64], Vinardo [65],
Smina [66], and AutoDock4 [67].
Results From the docking power benchmark results shown in Figure 3, Uni-Mol ranks the 1st,with the top 1 success rate of 91.6%.
For comparison, the previous top scoring function AutoDock
Vina [63, 64] achieves 90.2% of the top 1 success rate in this benchmark.
From the binding poseaccuracy results shown in Table 5, Uni-Mol also surpasses all other baselines.
Notably, Uni-Moloutperforms the second best method by 22.81% under the threshold of 2Å. This result indicates that
Uni-Mol can effectively learn the 3D information from both molecules and pockets, as well as theinteraction in 3D space of them.
Even without pretraining, Uni-Mol (denoted as Uni-Mol random ) isalso better than other baselines.
This demonstrates the effectiveness of Uni-Mol backbone, as iteffectively learns the 3D information by limited data.
In summary, by combining molecular and pocket pretraining models, Uni-Mol significantly outperforms the widely used docking tools in the protein-ligand binding tasks.
Related work
Molecular representation learning Representation learning on large-scale unlabeled moleculesattracts much attention recently.
SMILES-BERT [18] is pretrained on SMILES strings of moleculesusing BERT [4].
Subsequent works are mostly pretraining on 2D molecular topological graphs [23,11]. MolCLR [12] applies data augmentation to molecular graphs at both node and graph levels, using
[Figure]
[Table]
[Figure]
[Table]
a self-supervised contrastive learning strategy to learn molecular representations.
Further, severalrecent works try to leverage the 3D spatial information of molecules, and focus on contrastive ortransfer learning between 2D topology and 3D geometry of molecules.
For example, GraphMVP [26]proposes a contrastive learning GNN-based framework between 2D topology and 3D geometry.
GEM [13] uses bond angles and bond length as additional edge attributes to enhance 3D information.
As aforementioned, due to the inability of handling 3D information, most previous representationlearning models cannot be used in the important 3D prediction tasks.
SE(3)-Equivariant models In many-body scenarios such as potential energy surface fitting, SE-(3)equivariance is usually required.
A series of SE(3) models are proposed, such as SchNet [68], tensorfield networks [69], SE(3) Transformer [70], DimmNet [71], equivariant graph neural networks(EGNN) [36], GemNet [72] and SphereNet [73].
Most of these models are used in supervisedlearning with energy and force.
In Uni-Mol, based on the standard Transformer, we introduce severalminor changes to make the model SE(3)-Equivariant.
Pocket druggability prediction Druggability prediction of protein binding pockets is crucial fordrug discovery as druggable pockets need to be identified at the beginning.
Since proteins undergoconformation changes that might alter the druggability of pockets, it is necessary to utilize 3Dspatial data beyond sequential information.
Early methods, such as Volsite [74], DrugPred [59], and
PockDrug [75], predict druggability based on the predefined descriptors of pockets’ static structures.
Later, TRAPP-CNN [60], based on 3D-CNN, proposes the analysis of proteins’ conformation changesand the use of such information for druggability prediction.
Protein-ligand binding pose prediction In structure-based drug design, it is crucial to understandthe interactions between protein targets and ligands.
The in vitro estimation of the binding poseand affinity, such as docking, allows for lead identification and guides molecular optimization.
Inparticular, docking is one of the most important approaches in structure-based drug design and hasbeen developed for the past decades.
Tools such as AutoDock4 [67], AutoDock Vina [63, 64], and Smina [66] are among the most used docking programs.
Also, machine learning-based dockingmethods, such as ∆ V ina RF 20 [76], DeepDock [61] and Equibind [77], have also been developed topredict protein-ligand binding poses and assess protein-ligand binding affinity.
Conclusion
In this paper, to enlarge the application scope and representation ability of molecular representationlearning (MRL), we propose Uni-Mol, the first universal large-scale 3D MRL framework.
Uni-Molconsists of 3 parts: a Transformer based backbone to handle 3D data; two large-scale pretrainingmodels to learn molecular and pocket representations respectively; finetuning strategies for all kindsof downstream tasks.
Experiments demonstrate that Uni-Mol can outperform existing SOTA invarious downstream tasks, especially in 3D spatial tasks.
There are 3 potential future directions.
1) Better interaction mechanisms for finetuning two pretrainingmodels together.
As the interaction between the pretraining pocket model and the pretrainingmolecular model is simple in the current version of Uni-Mol, we believe there is a large room forfurther improvement.
Large Uni-Mol models. As larger pretraining models often perform better, itis worthy of training a large Uni-Mol model on a bigger dataset. 3) More high-quality benchmarks.
Although there have been many applications in the field of drug design, high-quality public datasetshave been lacking.
Many public datasets cannot satisfy real-world demand due to the low data quality.
We believe the high-quality benchmarks will be the lighthouse of the entire field, and will significantlyaccelerate the development of drug design.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
[Table]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
[Table]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] The data we used are all from public databases and details in data processing are explained in Appendix.
The data, code, [Table] and instructions will be made public upon the acceptance of the paper.
[Table] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] We report all the training details for the experiemnt in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report the mean and std for different runs of experiments in Table 1,Table
and Table 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We report the detailed computing resources [Table] used for the experiment in Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We discuss all the used [Table] datasets in the experiment section 3, datasets and setup part.
(b) Did you mention the license of the assets?
[Yes] We mention the license for the datasets used in Appendix.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on
participant compensation?
[N/A]