Zero-Shot 3D Drug Design by Sketching and Generating
Abstract
Drug design is a crucial step in the drug discovery cycle.
Recently, various deep learning-based methods design drugs by generating novel molecules from scratch, avoiding traversing large-scale drug libraries.
However, they depend on scarce experimental data or time-consuming docking simulation, leading to overfitting issues with limited training data and slow generation speed.
In this study, we propose the zero-shot drug design method D ESERT ( D rug d E sign by S k E tching and gene R a T ing).
Specifically, D ESERT splits the design process into two stages: sketching and generating, and bridges them with the molecular shape.
The twostage fashion enables our method to utilize the large-scale molecular database to reduce the need for experimental data and docking simulation.
Experiments show that D ESERT achieves a new state-of-the-art at a fast speed.
Introduction
Drug design is a crucial step in the drug discovery cycle, which is the inventive process of finding new drugs based on a biological target (usually a protein pocket) [1, 2, 3].
However, seeking appropriate drugs for a particular target is quite challenging due to the enormous space of drug candidates (almost 10 33 ).
Traditional drug design approaches usually employ virtual screening [4, 5, 6] and molecular dynamics [7, 8] to traverse in a large scaled drug library, which is time-consuming and could not produce novel drug candidates.
Recently, a line of work proposes to realize drug design by generating drug molecules from scratch using deep generative models [9, 2, 10], which is quite promising due to the fast speed and the ability of de-novo drug design.
Most of current drug generation model are developed upon 1D (SMILES) [11, 12, 13] or 2D (molecular graph) [14, 15, 16, 17, 18, 19] molecular structures, which heavily rely on expensive experimental data for supervised training while ignoring the 3D interaction information between the drug and the pocket.
In a word, they attempt to find a molecule that maximizes the score given by a bio-activity predictor trained on experimental data.
They employ different optimization methods, such as Generative
Adversarial Network (GAN) [20], Bayesian Optimization (BO) [21, 22, 9], Reinforcement Learning (RL) [23, 24, 20, 25, 26, 27], Evolutionary and Genetic Algorithms (EA/GA)
[28, 29, 30, 31, 32], and Markov Chain Monte Carlo (MCMC) [33, 34], in the molecular space for obtaining desired drug molecule under certain constraints.
However, we argue that a purely data-driven approach of drug design is practically limited, since in most cases, the quantity of experimental drug-pocket pairs hardly enables supervised training of de-novo drug design.
Generally, most protein pockets lack bio-activity data, and learning on noisy and deficient data may lead to severe overfitting problems.
[Figure]
does not change too much an less than complemen
[Figure]
Generate
Recently, several drug design models have been proposed to directly generate drug molecules in the 3D space, the realistic space of drug-target interaction.
Generating in such space is very promising for the potential of leveraging some prior knowledge (e.g., physical knowledge) instead of entirely counting on data-driven methodology.
Specifically, Masuda et al.
[35] and Luo et al.
[2] propose efficient 3D generative models to learn the atom density conditioned on protein pockets, with GAN and auto-regressive models, respectively.
Nevertheless experimental data are still obligatory in their models for achieving satisfactory results.
Even more noteworthy is GEKO [10], which combines the ideas of 3D generation and physical simulation to obtain state-of-the-art drug design performance without the help of large-scale experimental data.
Intuitively, GEKO performs geometric editing in the 3D molecule space guided by docking simulation (physical knowledge) [36, 37].
However,
GEKO may suffer from two concerns: a) the frequent invocation of docking is very time consuming, which significantly slows down the speed of the drug design model [38, 39].
b) docking may not always be accurate enough, especially in some complex settings [40, 41].
In such a case, being heavily dependent on the docking accuracy could hurt the generalization of the proposed drug design model.
In this paper, we propose the zero-shot approach D ESERT , namely D rug d E sign by S k E tching and gene R a T ing.
Motivated by the idea of structure determines properties [42, 43, 44, 45], D ESERT is built on the assumption that molecular shape determines bio-activity between drug molecules and its target pocket.
In other words, we suppose that a drug candidate would have satisfactory bio-activity to a target pocket if their shapes are complementary (see Figure 1).
With such prior, as shown in Figure 2, D ESERT splits the whole drug design process into two stage: sketching and generating, which employs the molecular shape as the bridge of the two stages.
Such splitting makes D ESERT enjoy two advantages: a) D ESERT does not heavily rely on docking simulation, which only optionally uses docking for post-process and thus avoids the aforementioned disadvantages of GEKO.
b) D ESERT abandons the expensive experimental data.
Specifically, in the sketching stage, we only need to sample some reasonable shapes complementary to the target pocket.
In the generating stage, D ESERT proposes to employ a generative pre-trained model (from shape to concrete molecular) to fill the shape obtained in the last sketching stage.
Notably, the generative pre-trained model is only trained on the ZINC database, which contains 1000M pairs of molecules and their corresponding shapes.
This process does not rely on experimental data, making
D ESERT work in a zero-shot fashion.
Note that D ESERT is not baseless.
Besides the idea that structure determines properties (an important concept of Structural Biochemistry), we also have preliminary results to verify our assumption.
Figure 1 shows that after binding to a pocket, the root-mean-square deviation (RMSD) is not too shabby ( 1 . 391 Å) compared with molecular conformation generation methods such as CGCF [46] ( 1 . 248 Å) and ETKDG [47] ( 1 . 042 Å).
The results suggest that the molecular shape is stable when molecules bind to proteins.
We also show in Figure 1 that a ligand often attaches tight to a pocket, which means their shape is complementary.
In summary, our contributions are three-fold: (1) We propose D ESERT , a generative method for de-novo 3D shape-based drug design in the zero-shot setting.
(2) D ESERT trains a pre-trained model
Training
Reconstructed Molecules …
S h a p e E n c o d e r 3 D M o l e c u l e
D e c o d e r
Shape2Mol MoleculeShapes
[Figure]
molecules
Sketching
S h a p e E n c o d e r 3 D M o l e c u l e
D e c o d e r
Shape2Mol
Generating
DesignedMolecule
Designing (Ligand-based)
[Figure]
Sketching
S h a p e E n c o d e r 3 D M o l e c u l e
D e c o d e r
Shape2Mol
Generating
DesignedMolecule
Designing (Pocket-based)
[Figure]
from massive unbound molecules 3 , eliminating the constraints of labeled data.
(3) D ESERT achieves a new state-of-the-art result (the docking score improves 0 .
79kcal / mol and 2 .
93kcal / mol over the best supervised method on two datasets) at a fast speed (about 20 times faster than GEKO 4 ).
Proposed Method: D ESERT
In this section, we describe the proposed method in detail.
Inspired by the two aforementioned preliminary studies (see Figure 1 and Figure 2), D ESERT designs drugs for proteins in a two-stage fashion and employs the shape as a bridge since previous work has shown the feasibility of designing drug by molecular shape [49, 50, 51, 52, 53].
Specifically, D ESERT designs drugs by first sampling appropriate shapes complementary to the target pocket and then mapping the shapes to specific molecules.
In Section 2.1, we first introduce the overall picture of how D ESERT works in a zero-shot setting.
Then we pose two challenges: how to sketch the reasonable molecular shapes and how to generate corresponding molecules based on the shapes.
We put forward solutions in Section 2.2 and Section 2.3, respectively.
2.1 Zero-Shot Pipeline
In this section, we focus on how D ESERT designs drugs in the zero-shot setting.
Briefly speaking,
D ESERT produces molecules in a two-stage fashion: sampling the shape of the desired drug first (sketching) and generating molecules conditioned on the resulting shape (generating).
Zero-Shot Sketching There are mainly two cases when D ESERT needs to sample molecular shapes.
In the zero-shot case where no reference protein ligands are available, D ESERT samples reasonable shapes from protein pockets (see Figure 3c) based on biological observations.
Besides, D ESERT can also reuse the shape of a ligand to design a novel one (see Figure 3b).
Details are listed in Section 2.2.
Zero-Shot Generating D ESERT generates molecules through a pre-trained generative model, namely
S HAPE 2M OL , which can convert a given shape into diverse molecules (see Figure 3a).
In this procedure, we utilize massive unbound molecules to train the model.
Thus no information about proteins is needed.
Details of this model are presented in Section 2.3.
2.2 Sketching Molecular Shapes
The sketching stage is responsible for deciding what desired molecules look like.
In this section, we show how we design two heuristic methods to sketch the shapes of desired molecules.
[Figure]
There are mainly two cases when we sketch a molecule shape based on whether the ligand is provided.
When the ligand is available, the sketching process can be trivial since molecules with similar shapes have similar properties.
It is reasonable to directly use the ligand’s shape as the shape of the desired molecule, which we call Ligand-based Sketching .
If the ligand of the pocket is unavailable, the challenge is how to obtain a shape that has a high potential to bind to a pocket.
5 We name this Pocketbased Sketching , and our main idea is to sample a region with an appropriate size complementary to the surface of the pocket.
Our idea for Pocket-based Sketching is based on two main observations:
Ligands mainly lie in the area close to the pocket surface.
Figure 1 shows that the shape of satisfactory ligand is tightly complementary to the pocket.
Pockets are usually much larger than ligands, suggesting that directly utilizing the shape of pockets to design molecules is inappropriate.
To this end, we present an algorithm (see the Appendix 1.1) to obtain the desired molecule shape, which is of the appropriate size and complementary to the pocket surface.
We achieve this goal by finding another shape (namely seed shape) that intersects with the pocket, where the intersection has a similar size to a molecule.
We also show a 2D illustration in Figure 4.
2.3 Generating 3D Molecules by S HAPE 2M OL
In this section, we introduce S HAPE 2M OL , an encoder-decoder network mapping a shape to diverse and high-quality 3D molecules.
There are plenty of unbound molecule data, e.g., 1000M molecules in the ZINC database, making it possible to learn a large-scale pre-trained generative model from shape to molecule.
Concretely, we formulate the problem as an image-to-sequence generation, where the shape is voxelized as a 3D image (see 2.3.1), and the 3D molecule is converted to be a sequence (see 2.3.2).
Our generative approach is capable of modeling any complicated molecule structure and the linearization makes large-scale pre-training easier to implement.
2.3.1 Encoder: Voxelized Shape
The shape encoder is a 3D extension of the ViT [56], where we use 3D patches instead of 2D patches in the original ViT. Let A denote the set of all atoms, a molecule m can be constructed as a collection of atoms and their corresponding coordinates:
[Equation]
Given a molecule m , we transform its shape into a 3D image with a voxelization function v m : Z 3 → { 0 , 1 } :
[Equation]
where r denotes the Van der Waals radii [57], ϵ is a perturbed noise which helps prevent overfitting [58].
[BOS]
T
T
T… …
T n
T nT n [EOS]
T[BOS], T 1 , T 2 , [BOB], T 3 , [EOB], [BOB], T 4 , T 5 , [EOB], T 6 , T 7 , T 8 , [EOS]
T T
T T
T
T 7 T
Molecule
Tree Structure
Fragment Sequence
Tokenization
Linearization
2.3.2 Decoder: Linearized Molecule
The molecule decoder in our model is similar to the Transformer decoder in machine translation [59].
The main difference is that the decoding object here is a 3D molecule instead of a 1D sequence.
To address this, we propose a 3D molecule decoder, which handles a 3D molecule as a sequence of tuples.
The sequence object eases the implementation of a pre-trained model.
To obtain the object, we first cut a molecule into pieces, then convert it to a sequence.
Tokenization We cut a molecule into pieces so that the generative process can be easily factorized.
Our principles are three folds: (i) preserving the functional groups since they are vital for determining molecule properties [60], (ii) avoiding too large size of the vocabulary to ease the pre-training process [61], (iii) no circles exist in the segmented molecules since a tree structure is simpler to handle than a graph.
Our method is simple yet efficient: first tokenizing the molecules with BRCIS [62] and then cutting all single bonds attached to a ring.
More details about some pilot experiments can be found in
Appendix 1.2.
Linearization For the network output, we propose to utilize a linearized sequence to represent the target molecule graph, which is not only convenient for training, but also has the strong power to represent any complicated tree structure.
We first select an fragment whose degree is 1 as the root of the tree (e.g., T1 , T3 , T5 , and T8 in Figure 6).
Then we traverse the tree in the depth-first-traverse style[63].
Whenever we enter or leave a branch, we will add two special symbols [BOB] and [EOB] (beginning/ending of a branch), respectively.
More particularly, we use a tuple ( C, P, R ) to represent a fragment F , where C = 1 F is an indicator function [64] denoting its index in the vocabulary, P ∈ R 3 is the translation vector [65],
R ∈ Ris the rotation quaternion [66].
In order to stabilize the training process, we further discretize the continuous variable P and R into P c and R c , respectively.
Taking the translation vector P as an example, we convert it into a binary vector P c , which satisfies:
[Equation]
where L is the max translation length, b is the bin size.
2.4 Training & Decoding
Training Given the output probabilities of the model ( ˆ C i , ˆ P ci , ˆ R ci ) , which denote the probability of a fragment, a discreterized translation vector, and a discreterized rotation vector, respectively.
We calculate the corresponding cross-entropy loss and use their sum as the final loss function.
[Equation]
where n is the length of fragment sequence.
We use 100M unbound molecules sampled from the lead-like subset of ZINC as the training data.
The
Transformer’s dimension is 1024, and both encoder and decoder have 12 stacked Transformer layers.
When training S HAPE 2M OL , we set the dropout rate as 0.1, batch size 2048, train step 300K and use
AdamW [67] with learning rate 5e-4, weight decay 1e-2, and warmup step 4000 as the optimizer.
The model is developed by ParaGen [68] 6 and trained on 32 Telsa V100 GPU cards for 2 weeks.
Following [35], we also randomly rotate and translate the input shape for invariance.
Decoding We design a decoding strategy to provide diverse and high-quality candidate molecules for a given protein.
To achieve diversity, we employ the sampling method Nucleus [69] to generate multiple fragment sequences.
Then we convert the sequences back to molecules with a greedy algorithm (see Appendix 1.2), which connects the fragments by greedily enumerating the nearest pair of breakpoints.
Finally, we do some post-processing operation to further improve the diversity and quality.
We remove the duplicate molecules and leverage the docking simulation to drop molecules that do not pass the affinity threshold.
When testing S HAPE 2M OL , we set the threshold of Nucleus sampling to 0.95.
For each protein pocket, we sketch 200 shapes.
For each shape, we generate 1000 molecules.
More details of
S HAPE 2M OL can be found in the Appendix 1.3.
Results and Discussions 3.1 Experiments Data
We evaluate the performance of our method on drug design by using a total of 12 proteins (PDB IDs: 1FKG, 2RD6, 3H7W, 3VRJ, 4CG9, 4OQ3, 4PS7, 5E19, 5MKU, 3FI2, 4J71), which is a combination of the test data used in Masuda et al.
[35] and Jin et al.
Among the proteins, only JNK3 and GSK3 β (PDB IDs: 3FI2, 4J71) have sufficient labeled data to train well-performing bioactivity predictors.
We denote the set of these two proteins as Set B and the rest as Set A.
Because 1D/2D methods need bioactivity predictors to design drugs for given proteins, we only evaluate them on Set B while evaluating 3D-based methods on both sets.
Baselines
We compare D ESERT with some baselines for drug design.
Based on the resource needed for designing drugs, we further divide these baselines into three groups: Guided methods need docking simulation or extra bioactivity predictors to provide supervision signals.
Supervised methods rely on labeled data to train their models.
Retrieved methods directly search the database for desired molecules.
Evaluation
Following Yang et al.
[10], we evaluate the performance of methods from two aspects.
(1) the molecular space covered by designed results.
(2) the capacity to provide highly active molecules.
As a high-quality molecular space should contain diverse, novel molecules with pharmaceutical potential, we use five metrics to evaluate the molecular space: Uniqueness ( Uniq ),
Novelty ( Nov ), Diversity ( Div ), Success rate ( Succ ), and Product ( Prod ).
To evaluate the ability to provide highly active molecules, we compare the distributions of Vina score and use Median Vina
Score ( Median ) to quantify the distribution.
More details about these metrics can be found in
Appendix 2.1.
Detail of Generation We describe how different methods generate molecules for comparison.
For each protein, every method needs to generate 100 molecules for comparison.
For 3D methods,
we
[Table]
further use the local minimization module in Vina [36] to optimize the generated structures.
For 1D/2D methods
[Table]
we first use RDKit to generate the 3D conformer of their results.
For method
S CREEN , we sample 1K and 200K molecules from ZINC and return 100 molecules with the highest
Vina score as the generated results.
3.2 Main Results
Table 1 shows the results of D ESERT and baselines.
All values are averaged over target proteins.
Our main findings are listed as follows:
I.The zero-shot D ESERT achieves the SOTA result at a fast speed.
D ESERT ’s performance is strong compared with GEKO, an MCMC-based model with a huge sampling space.
Compared with GEKO, which works in a trial-and-error way, D ESERT makes a more clever choice by pruning the space with its biological knowledge regarding the shape and quickly finds a good solution with limited hints from a teacher.
We compare the generation speed in Figure 7.
II.The shape helps D ESERT produce high quality molecules.
The molecular space of S CREEN is the ZINC database, while that of D ESERT is generated by the pre-trained model S HAPE 2M OL , which is aware of the pocket’s shape.
According to the table, both D ESERT -L IGAND and D ESERTP OCKET show observably better performance than their counterparts, i.e., S CREEN (1K) and S CREEN (200K).
III.
More comprehensive exploration of protein pockets benefits performance
Instead of using the molecular shapes of reference ligands as input, D ESERT -P OCKET comprehensively explores the protein pockets by sampling multiple molecular shapes from them.
Therefore, it has the potential to obtain diverse, high-quality molecules that bind to protein pockets in different regions.
While D ESERTL IGAND only considers one region, i.e., the region that ligands lie in, limiting the exploration of protein pockets.
We show a case in Figure 8.
[Figure] [Figure]
IV.
Unsupervised methods have larger potential than the supervised counterparts.
As labeled data is inadequate, e.g., scPDB 8 only has 16,034 entries, the supervised methods easily collapse to the main molecule pattern in their dataset.
When generating molecules with them, we often get the same molecules, e.g., 3D SBDD can only generate 16 unique molecules for protein 4OQ3.
D ESERT utilizes massive unbound molecules, which leads to the learned space being denser.
Combing with an appropriate sampling method, it can generate diverse molecules.
3.3 Comparison with Related Shape-based Models
In this section, we study the shape faithfulness and structure rationality compared with previous shape-based models.
In Table 2, we use Shape Tanimoto 9 to evaluate the faithfulness and use
Free Energy to quantify the rationality [74].
We compares D ESERT -L IGAND with liGAN (Ligand), a variant of liGAN [35] which utilizes the existing ligands.
Although liGAN (Ligand) achieves high performance on Shape Tanimoto, its atom-based decoding strategy does not guarantee the correct relative position between atoms.
Therefore, liGAN shows higher Free Energy, indicating that unrealistic structures may appear.
Figure 9 shows a case where liGAN produces an ill ring structure.
3.4 Ablation Study of Generating
In this section, we evaluate several designs of the generating stage in our D ESERT method, which relate to the pre-trained model and decoding strategy.
All results are based on an extra test set from
ZINC and a smaller version of S HAPE 2M OL .
Pre-training Configuration We evaluate the model quality on different pre-training configurations (mainly focusing on the size of the model and training data).
The results in Figure 10 show: (1)
Larger model achieves better performance.
Devlin et al.
[75] observes similar phenomenons in natural language modeling.
(2) Performance saturation occurs when the dataset is of moderate size.
As the
[Figure] [Figure]
[Figure]
[Table]
map from radii to atoms is easy to learn, the model can capture it with a moderate dataset.
Liu et al.
[76] reports a similar result that a large dataset does not necessarily lead to better quality.
More Ablation We also do ablation study about some model variants (including discretization and robust training ) and decoding strategies .
We further study chemical information driven design and atom-based pre-training .
We refer the reader for more details to the Appendix 2.2.
3.5 Ablation Study of Sketching
In this section, we study the sketching stage in our D ESERT method, which includes the effect of sampling space size and seed shape on the method’s performance.
All the results are based on the same D ESERT -P OCKET method in Section 3.2 and are calculated on Set B.
Sampling Space Size
In Figure 11, we evaluate the performance of D ESERT with respect to sampling space size, i.e., the total number of generated molecules before post-processing.
The results show: (1) Increasing sampling space size leads to better performance.
With a larger sampling space,
D ESERT finds more molecular shapes complementary to pockets, leading to a performance rise.
The shape can effectively prune the sampling space for screening.
Instead of directly searching molecular space, D ESERT achieves a similar performance by pruning the space from 200K to 10K with molecular shapes.
More Ablation We also do ablation study about the usage of different seed shape .
We refer the reader for more details to the Appendix 2.3.
3.6 Apply D ESERT to More Protein Targets
To test the generalization ability of our method more widely, we also apply D ESERT to the test data from Luo et al.
[2], which contains 100 protein targets.
As shown in Table 3, based on the idea of structure determines properties , D ESERT -P OCKET generalizes well on different target proteins in both setting, i.e., with/without post-processing.
Supervised methods, like 3D SBDD, hindered by scarce training data, can not generate diverse molecules.
In contrast, training on massive unbound and drug-like molecules, D ESERT easily generates diverse and promising molecules.
Moreover, sketching molecular shapes based on given pockets also contributes to the better binding affinity of
D ESERT .
However, DESERT gives a lower SA score than 3D SBDD.
We assume that it is because the generated molecules of DESERT tend to be structurally complicated, which leads to a slightly worse synthesis score.
Conclusions
In this study, we propose a zero-shot drug design method D ESERT , which splits the drug design process into two stages: sketching and generating.
D ESERT bridges the two stages with the molecular shape and utilize a large-scale molecular database to reduce the dependence on experimental data and docking simulation.
Experiments show that D ESERT achieves a new state-of-the-art at a fast speed.
Acknowledgements
We would like to thank the anonymous reviewers for their insightful comments.
Both Hao Zhou and Xinyu Dai are the corresponding authors.
This work is jointly supported by Guoqiang Research
Institute General Project, Tsinghua University (No. 2021GQG1012) and National Science Foundation of China (No. 61936012 and 61976114).
Method 1.1 Sketching
In Pocket-based Sketching, we present an algorithm to obtain the molecule shape, which is of the appropriate size and complementary to protein pockets.
The algorithm uses a seed shape to intersect with the pocket gradually.
To obtain the seed shape, we first randomly sample several molecules from ZINC, then use the overlapping of their shapes as the seed shape.
By using the overlapping strategy, we make the sketched pseudo molecular shapes more native-molecule-like and not overly dependent on one specific molecule.
Algorithm 1 is the pseudo-code.
We set the volume threshold to 300 Å 3 , the average volume of molecules.
The step size is the same as the voxel resolution, i.e., 0 .
5 Å. We initially place the seed shape in a random position as long as the seed shape and the pocket shape do not overlap.
1.2 Generating
Pilot Experiments of Tokenizing Methods We evaluate different molecule tokenizing methods based on three principles (preserving functional groups, appropriate vocabulary size, and no circles structures) and propose our approach in the context of pre-training.
There are several popular tokenizing methods for drug design: Atom-based [1, 2] shatters a molecule into atoms.
Cut Single
Bond [3] cuts all single bonds in a molecule.
Both RECAP [4] and BRICS [5] use chemical reaction rules for preserving functional groups.
The results in Table 1 show: (1) Atom-based and Cut Single
Bond can not preserve functional groups.
Because they do not distinguish whether a chemical bond belongs to a functional group, they ruin the molecular structures that are vital for determining molecule properties.
(2) Two chemical-reaction-aware methods behave differently.
As RECAP has more conservative rules, it produces a vocabulary at least one magnitude larger than its counterparts.
Because BRICS shows a better balance among these principles, we adapt it to serve our pre-trained model.
Furthermore, because over 60% of the out-of-vocabulary problem, an essential factor affecting the pre-trained model quality [6], is caused by the combinations between rings and other structures through single bonds, we add an extra rule to BRICS: cut all single bonds attached to a ring.
Details of Discretization To stabilize the training process, we discretize two continuous variables, i.e., translation vector and rotation quaternion.
Specifically, we map them into two discrete spaces.
For the translation vector, its discrete space is grids in 3D space (see Figure 1a).
Briefly speaking, we represent this continuous vector with the discrete index of the grid.
We represent the i -th translation bin as the coordinate of its centre t bin i
[Equation]
The discretization of any continuous translation operator t ∈ R 3 can be computed by arg min i
[Equation]
The total number of grids is 21 , 952 .
Because the rotation axis and rotation angle can determine the rotation quaternion, the discrete space of the quaternion contains two parts.
As shown in Figure 1b, we first enumerate rotation axes ( x, y, z ) in 3D space, then for each axis, we list the rotation angle every θ degrees.
We represent the i -th rotation bin (stands for rotating θ i degrees around an axis ( x i , y i , z i ) ) as a quaternion 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Input: a function λ measuring the volume of a shape, a threshold t , a step size α a pocket shape S p located at C p , a seed shape S s located at C s where λ (
T ( S p , S s )) =
Output: The sampled molecule shape S d . while True do
V ← λ (
T ( S p , S s )) if V ≥ t then S d ←
T ( S p , S s ) break end if
C s = αC p + (1 − α )
C s end while
[Table]
= ( x i , y i , z i , θ i )
∈
R 4 .
The discretization of any continuous rotation operator q ∈ R 4 can be computed by arg min i ∥
q bin i , q ∥ 2 .
The total number of bins is 8 , 763 .
To be precise, we enumerateaxes and 24 angles (i.e., 15 degrees per angle).
Another reason for the discretization is to avoid the discontinuity of quaternions when optimizing them [7, 8].
Specifically, there are several approaches to parameterize the rotation operator: quaternion [9], euler-angle [10], and SO(3) group (i.e., the rotation matrix)
Quaternion and euler-angle are sometimes ambiguous and discontinuous.
For example, the rotation operator is periodic, rotating 180 ◦ is equal to rotating − 180 ◦ and rotating 179 .
9 ◦ is very close to rotating − 179 .
9 ◦ .
Without discretization, we will excessively penalize the model according to the mean-square-error (i.e., [179 . 9 − ( − 179 . 9)] 2 = 359 . 8 2 ).
While in this paper, we convert a regression problem into a classification one with discretization, thus avoiding the above issues.
Instead of discretizing quaternions, AlphaFold [12] avoids such issues by regarding the quaternion as an intermediate variable and not optimizing the quaternion directly.
We leave adopting AlphaFold’s strategy for future work.
Greedy Algorithm for Connecting Fragments Algorithm 2 is responsible for converting the separated fragments into a complete 3D molecule.
To be specific, we first place all the fragments in 3D space according to the predictions of S HAPE 2M OL .
Then, for each time, we greedily chose the
[Figure]
𝜃 ° origin rotationaxis
[Figure]
two closest breakpoints from different fragments and connect the fragments through the breakpoints.
By repeating the process, the fragments get larger and larger.
When there are not enough breakpoints to connect, we return the largest fragment as the final molecule.
For potential residual breakpoints, we attach carbon atoms to them for molecular validity.
Algorithm
Input: Fragment sequence S =
[( C i , P i , R i )]
l s i =0 , fragment vocabulary V .
Output: A whole molecule M formed by the fragments.
B ← {} F ← {} for i ← 0 to l s do ( C i , P i , R i ) ← S [ i ] f ← query_vocabulary( V , C i ) f ← rotate(translate( f , P i ), R i )
▷ Place the fragment in 3D space b ← get_breakpoints( f ) ▷ Return all the breakpoints in current fragment
F ← F ∪ { f } B ← B ∪ { b } end for while | B | ≥ 2 do b 1 , b← get_breakpoints( B ) ▷ Chose two nearest breakpoints from different fragments if b 1 is None or b 2 is None then break end if f 1 ← get_fragment( F , b 1 ) ▷ Retrieve the corresponding fragment of the breakpoint f 2 ← get_fragment( F , b 2 ) M partial ← attach_fragments( f 1 , b 1 , f 2 , b 2 )
▷ Connect fragments through the breakpoints
B ← B \ { b 1 , b 2 } F ← F \ { f 1 , f 2 } F ← F ∪ { M partial } end while
M ← get_largest_fragment( F ) ▷ Chose the largest fragment as the output molecule if | B | ̸ = 0 then M ← attach_carbon( M , B ) ▷ Handle the remaining breakpoints by attaching carbon atoms end if
1.3 More Details of S HAPE 2M OL
Output of Shape Encoder
The output of the shape encoder is the continuous representation of each 3D patch, which contain the geometric information of input molecular shape.
It will serve as the context of the decoder to constrain the shape of generated molecules.
Input of Shape Decoder
The input of the shape decoder at the time step t is the fragment category, rotation quaternion, and translation vector from the output of the previous step t − 1 .
We use them to tell the model how exactly a fragment is placed in 3D space so that the model can generate the next fragment connected with it.
The output of the shape encoder is fed into the decoder as the geometric context, through the cross-attention module.
Hyperparameters of S HAPE 2M OL
Both encoder and decoder of S HAPE 2M OL have 12 stacked 8 -head Transformer layers, whose model size is 1024 and feedforward dimension is 4096 .
We use
ReLu as the activation function and set the position embedding learnable in the training process.
For the encoder, following liGAN, we set the resolution of the voxelized shape to 0 .
5 Å and the side length of the spanned cube to 14 Å.
The size of 3D patches is 4 × 4 × 4 , which allows us to handle a large number of voxels.
Specifically, we reshape 21 , 952 voxels into 343 3D patches in the paper.
The decoder has 3 different embeddings, denoting the fragment category, the discretized rotation, and the discretized translation, respectively.
Similarly, we also use 3 different output heads to predict the category, rotation, and translation.
The output heads share their weights with the embeddings.
The total number of model parameters is 650M.
Experiments 2.1 Metrics
• Uniqueness (Uniq) is the percentage of unique molecules among all generated results.
• Novelty (Nov) is the percentage of generated molecules with Tanimoto similarity [13] less than 0.4 compared to its nearest neighbor in existing ligands [14].
• Diversity (Div) is the internal diversity of the generated molecules and calculated as
[Equation]
′ 1 − sim ( x, x ′ ) .
• Success rate (Succ) is the percentage of generated molecules that pass the predefined thresholds for the desired properties.
Specifically, we based on a widely used rule of thumb [15] to set the thresholds as QED ≥ 0 .
25 , SA score ≥ 0 .
59 and Vina score ≤ − 8 .
18 kcal/mol, where QED is the index of drug-likeness [16] and SA score is the index of synthetic accessibility [17].
• Product (Prod) is the product of the four metrics above and serves as a comprehensive evaluation.
• Median Vina Score (Median) is the median of Vina score .
Vina score is the binding energy evaluation using the global energy optimization algorithm in Vina [18], which can reflect the binding affinity between proteins and molecules.
2.2 More Ablation Study of Generating
Model Variants As shown in Figure 3, we study two variants in the model design: discretization and robust training.
The discretization consistently improves the result, while the continuous variant is worse.
A possible reason is a non-linear relationship between quaternions and rotation angles.
We observe minor improvement for finer granularity and leave it for future exploration.
We also study the ability of handling shape noise caused by a possibly large pocket size.
The model trained with shape noise (i.e. robust training) shows better performance when test data are noisy, while the performance under standard training drops clearly.
[Figure] [Figure]
[Table] [Table]
[Table]
Decoding Strategy We replace the Nucleus sampling method, i.e., Top P sampling, in the decoding strategy with Beam Search [19], Greedy Decoding [20]
[Table]
and Top K sampling [20] to study the influence of different sampling methods on the performance of D ESERT .
As shown in Table 2, Beam
Search and Greedy Decoding are poor at providing diverse molecules, which suggests that the output probability is concentrated.
They can only get several fragments with high output probability.
When low probability fragments are obtained, the ranking mechanism in Beam Search drops them in the subsequent decoding step.
Due to their internal annealing mechanism, both Top K and Top P can provide relatively diverse results.
Chemical Information Driven Design We also explore the potential of integrating chemical information of proteins into drug design.
Briefly speaking, based on hydrogen bond acceptor-donor rules [21], we put the fragment with more hydrogen atoms into the pocket region with more oxygen, nitrogen, and fluorine atoms.
The results in Table 3 show that the strategy does not achieve better performance when we increase the effect of chemical information on the output probability.
The reason might be not considering detailed chemical information, like the bond length.
We leave a more thoughtful method for combining the chemical information for future work.
Atom-based Pre-training We pre
-train an atom-based S HAPE 2M OL to analyze the contribution of our fragment-based generation style.
Specifically, we keep most pre-training processes the same but replace our tokenizing method with the atom-based strategy (both mentioned in Section 1.2).
As drug molecules mainly consist of carbon atoms, the atom-based S HAPE 2M OL ignores input shapes and keeps outputting carbon atoms.
The atom-based variant can not design any valid molecules, which indicates the fragment-based style is crucial for our pre-training process.
Invariance of S HAPE 2M OL
Following liGAN, when training S HAPE 2M OL , we randomly rotate and translate the molecules to make our model have rotation and translation invariance ability.
We compare the similarity of generated molecules based on different or the same molecular shapes (both are randomly rotated and translated).
The similarity rises from 0.092 to 0.508, which shows that with the same molecular shape as input, the model produces similar molecules.
2.3 More Ablation Study of Sketching
Seed Shape Type We evaluate the influence of different seed shape in Table 4.
We find that directly using the pocket as the shape may contribute to suboptimal result in providing highly active molecules.
Through the sketching stage, we obtain more realistic and diverse molecular shapes.
These shapes benefit the molecular activity but result in more complex structures and lower synthetic accessibility in generated molecules.
Discussion of Related Work 3.1 Shape-based Drug Design
Here we discuss the relationship between our method and existing shape-based drug design [22] approaches.
Some previous work designed new drugs based on the shape of a known ligand.
Traditional approaches work in a retrieval way – finding molecules whose shape is most similar to a known one [23, 24].
Modern deep learning models can decode a molecule from its shape [1, 25].
Such ligand-based generation can not generalize to unseen pockets.
Luo et al.
[2] directly generates molecules from the pocket shape, which is mostly close to our work.
However, our model works in a fragment-based fashion, while theirs works in an atomic way.
What makes our model especially different is that we utilize the power of the pre-training model to make pocket-based drug design more promising.
3.2 Tokenization and Linearization
Here we discuss how the tokenization and linearization procedures relate to fragment-based drug design (FBDD) [26, 27].
Fragment-Based Drug Design Briefly, there are two approaches in FBDD: growing the fragment synthetically to a proximal binding site or by linking two fragments together [27].
Our method can be classified as the previous type since the linearization generates a molecule in a one-by-one fashion.
Tokenization The procedure is carefully designed for deep generative models to avoid loops and preserve functionalities, which is in spirit to the principle of [3].
Some FBDD work, such as [28] works in a discriminative approach.
Thus there are not many constraints when cutting the molecules.
Podda et al.
[29] uses the SMILES-fragment rather than a real molecule-fragment.
Thus it can not utilize rich structured features.
Linearization The procedure aims at traversing (or generating) a structured object in a left-toright approach [30], which is tractable and scalable.
It is borrowed from the area of computational linguistics, more specifically, syntactic parsing and structured generation [31, 32].
For micromolecules, linearization (such as SMILES [33]) has been adopted for several decades.
There is a line of research work for SMILES-based generation [34, 35, 36].
Similarly, in the area of macro-molecule,
Huang et al.
[37] designs a linear structure for estimating the likelihood of the structure of RNA.
Comparison Compared with traditional linearized sequences such as SMILES [33], our method utilizes structural information to segment the molecules to preserve their functionality.
Compared with topological generation based on a graph [3, 38, 39], our method is more scalable to big data since generating the variable graph topology is not friendly to large-batch training in neural networks.
More Cases
We show more cases from liGAN and D ESERT in Figure 2.
Similarly, liGAN struggles to produce realistic molecular structures, while D ESERT generates better results.