S PATIAL G RAPH A TTENTION AND C URIOSITY - DRIVEN P OLICY FOR A NTIVIRAL D RUG D ISCOVERY
Mikaela Cashman ¶ § cashmanmm@ornl.gov
Nicholas Choma ‡ § njchoma@lbl.gov ´Erica T. Prates ¶ § teixeiraprae@ornl.gov
Ver´onica Melesse Vergara || ¶ vergaravg@ornl.gov
Manesh Shah ||
shahmb@ornl.gov
Andrew Chen ‡ § adchen@lbl.gov
Austin Clyde †† aclyde@uchicago.edu
Thomas S. Brettin §§ brettin@anl.gov
Wibe A. de Jong ‡ § wadejong@lbl.gov
Neeraj Kumar ‡‡ § neeraj.kumar@pnnl.gov
Martha S. Head ¶ § headms@ornl.gov
Rick L. Stevens †† §§ stevens@anl.gov
Peter Nugent ‡ § penugent@lbl.gov
Daniel A. Jacobson || ¶ § jacobsonda@ornl.gov
James B. Brown † ‡ § jbbrown@lbl.gov
A BSTRACT
We developed Distilled Graph Attention Policy Network (DGAPN), a reinforcement learning model to generate novel graph-structured chemical representations that optimize user-deﬁned objectives by efﬁciently navigating a physically constrained domain.
The framework is examined on the task of generating molecules that are designed to bind, noncovalently, to functional sites of SARS-CoV-2 proteins.
We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure — this capability is of considerable interest in synthetic biology and drug discovery.
An attentional policy network is introduced to learn the decision rules for a dynamic, fragment-based chemical environment, and state-of-the-art policy gradient techniques are employed to train the network with stability.
Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation.
In experiments, our framework achieved outstanding results compared to state-of-the-art algorithms, while reducing the complexity of paths to chemical synthesis.
I NTRODUCTION
This work aims to address the challenge of establishing an automated process for the design of objects with connected components, such as molecules, that optimize speciﬁc properties.
Achieving this goal is particularly desirable in drug development and materials science, where manual discovery remains a time-consuming and expensive process (Hughes et al., 2011; Schneider et al., 2020).
However, there are two major difﬁculties that have long impeded rapid progress.
Firstly, the chemical space is discrete and massive (Polishchuk et al., 2013), presenting a complicated environment for an Artiﬁcial Intelligence (AI) approach to efﬁciently and effectively explore.
Secondly, it is not trivial to compress such connected objects into feature representations that preserve most of the information, while also being highly computable for Deep Learning (DL) methods to exploit.
We introduce Distilled Graph Attention Policy Network (DGAPN), a framework that advances prior work in addressing both of these challenges.
We present a Reinforcement Learning (RL) architecture that is efﬁciently encouraged to take innovative actions with an environment that is able to construct a dynamic and chemically valid fragment-based action space.
We also propose a hybrid Graph Neural
Network (GNN) that comprehensively encodes graph objects’ attributes and spatial structures in addition to adjacency structures.
The following paragraphs discuss how we addressed limitations of prior work and its relevance to antiviral drug discovery.
For more descriptions of key prior methodologies that we used as benchmarks in this paper, see Section 4.
Graph Representation Learning Despite their spatial efﬁciency, string representation of molecules acquired by the simpliﬁed molecular-input line-entry system (SMILES) (Weininger, 1988) suffers from signiﬁcant information loss and poor robustness (Liu et al., 2017).
Graph representations have become predominant and preferable for their ability to efﬁciently encode an object’s scaffold structure and attributes.
Graph representations are particularly ideal for RL since intermediate representations can be decoded and evaluated for reward assignments.
While GNNs such as Graph Convolutional Networks (GCN) (Kipf & Welling, 2016) and Graph Attention Networks (GAT) (Veliˇckovi´c et al., 2017) have demonstrated impressive performance on many DL tasks, further exploitation into richer information contained in graph-structured data is needed to faithfully represent the complexity of chemical space (Morris et al., 2019; Wang et al., 2019; Chen et al., 2020).
In this work, we made improvements to previous studies on attributes encoding and structural encoding.
For structural encoding, previous studies have covered adjacency distance encoding (Li et al., 2020), spatial cutoff (Pei et al., 2020) and coordinates encoding (Sch ¨ utt et al., 2017; Danel et al., 2020).
Our work presents an alternative approach to spatial structure encoding similar to
Gilmer et al.
(2017) which do not rely on node coordinates, but different in embedding and updating scheme.
Distinct from Danel et al.
and Chen & Chen (2021), we extended attentional embedding to be edge-featured, while still node-centric for message passing efﬁciency.
Reinforcement Learning
A variety of graph generative models have been used in prior work, predominantly Variational Autoencoders (VAE) (Simonovsky & Komodakis, 2018; Samanta et al., 2020; Liu et al., 2018; Ma et al., 2018; Jin et al., 2018) and Generative Adversarial Networks (GAN) (De Cao & Kipf, 2018).
While some of these have a recurrent structure (Li et al., 2018; You et al., 2018b), RL and other search algorithms that interact dynamically with the environment excel in sequential generation due to their ability to resist overﬁtting on training data.
Both policy learning (You et al., 2018a) and value function learning (Zhou et al., 2019) have been adopted for molecule generation: however, they generate molecules node-by-node and edge-by-edge.
In comparison, an action space consisting of molecular fragments, i.e., a collection of chemically valid components and realizable synthesis paths, is favorable since different atom types and bonds are deﬁned by the local molecular environment.
Furthermore, the chemical space to explore can be largely reduced.
Fragment-by-fragment sequential generation has been used in VAE (Jin et al., 2018) and search algorithms (Jin et al., 2020; Xie et al., 2021), but has not been utilized in a deep graph RL framework.
In this work, we designed our environment with the Chemically Reasonable Mutations (CReM) (Polishchuk, 2020) library to realize a valid fragment-based action space.
In addition, we enhanced exploration by employing a simple and efﬁcient technique, adapting Random Network Distillation (RND) (Burda et al., 2018) to GNNs and proposing surrogate innovation rewards for intermediate states during the generating process.
Antiviral Drug Discovery — A Timely Challenge
The severity of the COVID-19 pandemic highlighted the major role of computational workﬂows to characterize the viral machinery and identify druggable targets for the rapid development of novel antivirals.
Particularly, the synergistic use of DL methods and structural knowledge via molecular docking simulations is at the cutting edge of molecular biology — consolidating such integrative protocols to accelerate drug discovery is of paramount importance (Yang et al., 2021; Jeon & Kim, 2020; Thomas et al., 2021).
Here we experimentally examined our architecture on the task of discovering novel inhibitors targeting the
SARS-CoV-2 non-structural protein endoribonuclease (NSP15), which is critical for viral evasion of host defense systems (Pillon et al., 2021).
Structural information about the putative protein-ligand complexes was integrated into this framework with AutoDock-GPU (Santos-Martins et al., 2021), which leverages the GPU resources from leadership-class computing facilities, including the Summit supercomputer, for high-throughput molecular docking (LeGrand et al., 2020).
We show that our results outperformed state-of-the-art generation models in ﬁnding molecules with high afﬁnity to the target and reasonable synthetic accessibility.
P ROPOSED M ETHOD 2. E NVIRONMENT S ETTINGS
In the case of molecular generation, single-atom or single-bond additions are often not realizable by known biochemical reactions.
Rather than employing abstract architectures such as GANs to suggest synthetic accessibility, we use the chemical library CReM (Polishchuk, 2020) to construct our environment such that all next possible molecules can be obtained by one step of interchanging chemical fragments with the current molecule.
This explicit approach is considerably more reliable and interpretable compared to DL approaches.
A detailed description of the CReM library can be found in Appendix B.1.
The generating process is formulated as a Markov decision problem (details are given in Appendix
A).
At each time step t , we use CReM to sample a set of valid molecules v t +1 as the candidates for the next state s t +1 based on current state s t .
Under this setting, the transition dynamics are deterministic, set A of the action space can be deﬁned as equal to S of the state space, and action a t is induced by the direct selection of s t +1 .
With an abuse of notation, we let r ( s t +1 ) :
= r ( s t , a t ) .
2. S PATIAL G RAPH A TTENTION
We introduce a graph embedding mechanism called Spatial Graph Attention (sGAT) in an attempt to faithfully extract feature vectors h t ∈ R d h representing graph-structured objects such as molecules.
Two different types of information graphs constructed from a connected object are heterogeneous and thus handled differently in forward passes as described in the following sections.
See Figurefor an overview.
are aggregated at the end of each layer.
2.2. A TTENTION ON A TTRIBUTION G RAPHS
The attribution graph of a molecule with n atoms and e bonds is given by the triple ( A , N , E ) , where A ∈ { 0 , 1 } n × n is the node adjacency matrix, N is the node attribution matrix of dimension n × d n and E is the edge attribution matrix of dimension e × d e .
Each entry a ij of A is 1 if a bond exists between atom i and j , and 0 otherwise.
Each row vector n i of N is a concatenation of the properties of atom i , including its atomic number, mass, etc., with the categorical properties being one-hot encoded.
E is formed similar to N , but with bond attributes.
We denote a row vector of E as e ij if it corresponds to the bond between atom i and j .
We proceed to deﬁne a multi-head forward propagation that handles these rich graph information: let h n k ∈ R 1 ×
d hn denote a given representation for n k , h e ij ∈ R 1 ×
d he denote a representation for e ij , then the m -th head attention α mij from node j to node i ( i (cid:54) =
j ) is given by
[Equation]
where softmax j is the softmax score of node j ; (cid:107) is column concatenation; σ is some non-linear activation; W n,m ∈ R d
hn × d
wn , W e,m ∈ R d he ×
d we are the m -th head weight matrices for nodes and edges respectively;
att m ∈ R 1 × (2 d wn + d we ) is the m -th head attention weight.
The representations after a feed-forward operation are consequently given as follow:
[Equation]
where W h,m ∈ R (2 d wn + d we ) ×
d we ; n m is the total number of attention heads and aggr denotes an aggregation method, most commonly mean , sum , or concat (Hamilton et al., 2017).
We note that we have not found signiﬁcant difference across these methods and have used mean for all aggregations in our experiments.
In principle, a single-head operation on nodes is essentially graph convolution with the adjacency matrix ˆ A
= (cid:101)
A
+ I where (cid:101)
A is attention-regularized according to (1).
This approach sufﬁciently embeds edge attributes while still being a node-centric convolution mechanism, for which efﬁcient frameworks like Pytorch-Geometric (Fey & Lenssen, 2019) have been well established.
2.2. S PATIAL C ONVOLUTION
In addition to attributions and logical adjacency, one might also wish to exploit the spatial structure of an graph object.
In the case of molecular docking, spatial structure informs the molecular volume and the spatial distribution of interaction sites — shape and chemical complementarity to the receptor binding site is essential for an effective association.
Let G = (cid:0) d ij − 1 (cid:1) i,j ≤ n be the inverse distance matrix where d ij is the Euclidean distance between node i and j for ∀
i (cid:54) =
j , and d ii − 1 := 0 .
G can then be seen as an adjacency matrix with weighted “edge”s indicating nodes’ spatial relations, and the forward propagation is thus given by
[Equation]
where (cid:101) G is optionally sparsiﬁed and attention-regularized from G to be described below;
(cid:101) D = diag 1 ≤ i ≤ n (cid:110)(cid:80) nj =1 (cid:101) G ij (cid:111)
; H n is the row concatenation of { h n i } 1 ≤ i ≤ n ; W n ∈ R d hn × d wn is the weight matrix.
In reality, G induces O ( n ) of convolution operations on each node and can drastically increase training time when the number of nodes is high.
Therefore, one might want to derive (cid:101) G by enforcing a cut-off around each node’s neighborhood (Pei et al., 2020), or preserving an
O ( n ) number of largest entries in G and dropping out the rest.
In our case, although the average number of nodes is low enough for the gather and scatter operations (GS) of Pytorch-Geometric to experience no noticeable difference in runtime as node degrees scale up (Fey & Lenssen, 2019), the latter approach of sparsiﬁcation was still carried out because we have discovered that proper cutoffs improved the validation loss in our supervised learning experiments.
If one perceives the relations between chemical properties and spatial information as more abstract, G should be regularized by attention as described in (1), in which case the spatial convolution is principally fully-connected graph attention with the Euclidean distance as a one-dimensional edge attribution.
2. G RAPH A TTENTION P OLICY N ETWORK
In this section we introduce Graph Attention Policy Network (GAPN) that is tailored to environments that possess a dynamic range of actions.
Note that ρ ( ·| s t , a t ) is a degenerate distribution for deterministic transition dynamics and the future trajectory τ ∼ p ( s t +1 , s t +2 , . . .
| s t ) is strictly equal in distribution to a ∼ π ( a t , a t +1 , . . .
| s t ) , hence simpliﬁed as the latter in the following sections.
To learn the policy more efﬁciently, we let s t and v t share a few mutual embedding layers, and provided option to pre-train the ﬁrst n g layers with supervised learning.
Layers inherited from pretraining are not updated during the training of RL.
See Figure 2 for an overview of the architecture.
2.3. A CTION S ELECTION
At each time step t , we sample the next state s t +1 from a categorical distribution constructed by applying a retrieval-system-inspired attention mechanism (Vaswani et al., 2017):
[Equation]
where OHC { p 1 , . . .
, p n v } is a one-hot categorical distribution with n v categories; g t , g t +1 are the embeddings for s t and v t +1 acquired by the shared encoder; E Q , E K are two sGAT+MLP graph encoders with output feature dimension d k ; L final : R b × 2 d k
→ R b is the ﬁnal feed-forward layer.
Essentially, each candidate state is predicted a probability based on its ‘attention’ to the query state.
The next state is then sampled categorically according to these probabilities.
There could be a number of ways to determine stopping time T .
For instance, an intuitive approach would be to append s t to v t +1 and terminate the process if s t is selected as s t +1 .
In our experiments, we simply pick T to be constant, i.e. we perform a ﬁxed number of modiﬁcations for an input.
This design encourages the process to not take meaningless long routes or get stuck in a cycle, and enables episodic docking evaluations in parallelization (further described in Section 2.5).
Note that constant trajectory length is feasible because the maximum limit of time steps can be set signiﬁcantly lower for fragment-based action space compared to node-by-node and edge-by-edge action spaces.
2.3. A CTOR -C RITIC A LGORITHM
For the purpose of obeying causal logic and reducing variance, the advantage on discounted rewardto-go are predominantly used instead of raw rewards in policy iterations.
The Q-function and advantage function are expressed as
[Equation]
where γ is the rate of time discount.
The Advantage Actor-Critic (A2C) algorithm approximates E π [ Q π ( s t , a t ) | s t ] with a value network V ζ ( s t ) and
Q π ( s t , a t ) with r ( s t , a t ) + γV ζ ( s t +1 ) .
For a more detailed description of actor-critic algorithm in RL, see Grondman et al.
2.3. P ROXIMAL P OLICY O PTIMIZATION
We use Proximal Policy Optimization (PPO)
(Schulman et al., 2017), a state-of-the-art policy gradient technique, to train our network.
PPO holds a leash on policy updates whose necessity is elaborated in trust region policy optimization (TRPO)
(Schulman et al., 2015), yet much simpliﬁed.
It also enables multiple epochs of minibatch updates within one iteration.
The objective function is given as follow:
[Equation]
where r t ( θ ) =
π newθ ( a t (cid:12)(cid:12) s t ) (cid:14) π oldθ ( a t (cid:12)(cid:12)
s t ) and clip (cid:15) ( x )
=
min { max { 1 − (cid:15), x } , 1 + (cid:15) } .
During policy iterations, π new is updated each epoch and π old is cloned from π new each iteration.
2. E XPLORATION WITH R ANDOM N ETWORK D ISTILLATION
We seek to employ a simple and efﬁcient exploration technique that can be naturally incorporated into our architecture to enhance the curiosity of our policy.
We perform Random Network Distillation (RND) (Burda et al., 2018) on graphs or pre-trained feature graphs to fulﬁll this need.
Two random functions ˆ f ψ , f ∗ that map input graphs to feature vectors in R d r are initialized with neural networks, and ˆ f ψ is trained to match the output of f ∗ :
[Equation]
(9) where ˆ p next is the empirical distribution of all the previously selected next states, i.e. the states that have been explored.
We record the running errors in a buffer and construct the surrogate innovation reward as:
[Equation]
where m b and v b are the ﬁrst and second central moment inferred from the running buffer, clip η ( x )
= min { max {− η, x } , η } .
2. P ARALLELIZATION AND S YNCHRONIZED E VALUATION
Interacting with the environment and obtaining rewards through external software programs are the two major performance bottlenecks in ours as well as RL in general.
An advantage of our environment settings, as stated in Section 2.3.1, is that a constant trajectory length is feasible.
Moreover, the costs for environmental interactions are about the same for different input states.
To take advantage of this, we parallelize environments on CPU subprocesses and execute batched operations on one
GPU process, which enables synchronized and sparse docking evaluations that reduces the number of calls to the docking program.
For future experiments where such conditions might be unrealistic, we also provided options for asynchronous Parallel-GPU and Parallel-CPU samplers (described in
Stooke & Abbeel (2019))
in addition to the Parallel-GPU sampler used in our experiments.
E XPERIMENTS 3. S ETUP
Objectives We evaluated our model against ﬁve state-of-the-art models (detailed in Section 4) with the objective of discovering novel inhibitors targeting SARS-CoV-2 NSP15.
Molecular docking scores are computed by docking programs that use the three-dimensional structure of the protein to predict the most stable bound conformations of the molecules of interest, targeting a pre-deﬁned functional site.
For more details on molecular docking and our GPU implementation of an automated docking tool used in the experiments, see Appendix B.2.
In addition, we evaluated our model in the context of optimizing QED and penalized LogP values, two tasks commonly presented in machine learning literature for molecular design.
The results for this can be found in Appendix D.
Dataset For the models/settings that do require a dataset, we used a set of SMILES IDs taken from more than six million compounds from the MCULE molecular library - a publicly available dataset of purchasable molecules (Kiss et al., 2012) and their docking scores for the NSP15 target.
3. R ESULTS 3.2. S INGLE - OBJECTIVE OPTIMIZATION
The raw docking score is a negative value that represents higher estimated binding afﬁnity when the score is lower.
We use the negative docking score as the main reward r m and assign it to the ﬁnal state s T as the single objective.
For DGAPN, we also assign innovation reward to each intermediate state, and the total reward for a trajectory τ is given by r ( τ ) =
r m ( s T ) + ι ·
T (cid:88) t = r i ( s t ) (11) where ι is the relative important of innovation rewards [Table] for which we chose 0 . 1 and incorporated them with a 100 episode delay and 1,000 episode cutoff.
Detailed hyperparameter settings for
DGAPN can be found in Appendix C. We sampled 1,000 molecules from each method and showed the evaluation results in Table 1.
We note that we have a separate approach to evaluate our model that is able to achieve a − 7 .
73 mean and − 10 . 38 best docking score (see the Ablation Study paragraph below)
[Table]
but here we only evaluated the latest molecules found in training in order to maintain consistency with the manner in which GCPN and MolDQN are evaluated.
[Table]
In the result table, ordinary validity is checked by examining atoms’ valency and consistency of bonds in aromatic rings.
In addition, we propose adjusted validity which further deems molecules that fail on conformer generation (Riniker & Landrum, 2015) invalid on top of the ordinary validity criteria.
This is required for docking evaluation, and molecules that fail this check are assigned a docking score of 0.
We also provide additional summary metrics to help gain perspective of the generated molecules: Uniq. and Div. are the uniqueness and diversity (Polykovskiy et al., 2020); QED (Bickerton et al., 2012) is an indicator of drug-likeness, SA (Ertl & Schuffenhauer, 2009) is the synthetic accessibility.
QED is better when the score is higher and SA is better when lower.
Deﬁnitions of QED and SA can be found in Appendix E.
On this task, DGAPN signiﬁcantly outperformed state-of-the-art models in terms of top scores and average score, obtaining a high statistical signiﬁcance over the second best model (MolDQN) with a p-value of 8 .
55 × 10 −under Welch’s t-test (Welch, 1947).
As anticipated, the molecules generated by fragment-based algorithms (JTVAE, MARS, DGAPN) have signiﬁcantly better SAs.
Yet we note that additional summary metrics are not of particular interest in single-objective optimization, and obtaining good summary metrics does not always indicate useful results.
For example, during model tuning, we found out that worse convergence often tend to result in better diversity score.
There also seems to be a trade-off between docking score and QED which we further examined in Section 3.2.3.
Ablation study We performed some ablation studies to examine the efﬁcacy of each component of our model.
Firstly, we segregate spatial graph attention from the RL framework and examine its effect solely in a supervised learning setting.
We performed 40 runs with independent samples of size 100,000 from the NSP15 dataset, and presented the result in Figure 3.
Secondly, we run single-objective optimization with (DGAPN) and without (GAPN) innovation rewards, and thirdly, compared the results from DGAPN in evaluation against greedy algorithm with only the CReM environment.
The results are shown in Table 2.
We note that it is not exactly fair to compare greedy algorithm to other approaches since it has access to more information (docking reward for each intermediate candidate) when making decisions, yet our model still managed to beat it in evaluation mode (see Appendix C for more information).
[Table]
In these experiments, spatial convolution exhibited a strong impact on molecular graph representation learning.
From results generated by the greedy approach, we can see that the environment and the stochasticity design of action space alone are powerful for the efﬁcacy and exploration of our policies.
While the innovation bonus helped discover molecules with better docking scores, it also worsened SA.
We further investigated this docking score vs. SA trade-off in Section 3.2.3.
To see samples of molecules generated by DGAPN in evaluation, visit our repository † .
3.2. C ONSTRAINED OPTIMIZATION
The goal of constrained optimization is to ﬁnd molecules that have large improvement over a given molecule from the dataset while maintaining a certain level of similarity:
[Equation]
(12) where λ is a scaling coefﬁcient, for which we chose 100 ; SIM {· , ·} is the Tanimoto similarity between Morgan ﬁngerprints.
We used a subset of 100 molecules from our dataset as the starting molecules, chose the two most recent and best performing benchmark models in single-objective optimization to compete against, and evaluated 100 molecules generated from theirs and ours.
The results are shown in Table 3.
[Table]
From the results
[Table]
it seems that MARS is not capable of performing optimizations with similarity constraint.
Compared to MolDQN
[Table]
DGAPN gave better improvements across all levels of δ
[Table]
although
MolDQN was able to produce molecules with more stable similarity scores.
3.2. M ULTI - OBJECTIVE OPTIMIZATION
We investigate the balancing between objective and realism by performing multi-objective optimization [Table] and thus provide another approach to generate useful molecules in practice.
We weight r m with a function of the two metrics, f ( QED, SA ) .
The choice of f and the new main reward are given by
[Equation]
where µ 1 and µ 2 are scaling coefﬁcients, for which we chose 8 and 9 respectively.
A summary of the results obtained from different ω are shown in Figure 4.
A trade-off between docking reward and QED/SA was identiﬁed.
We acknowledge that optimizing docking alone does not guarantee ﬁnding practically useful molecules, but our goal is to generate promising chemicals with room for rational hit optimization.
We also note that commonly used alternative main objectives such as pLogP and QED are themselves unreliable or undiscerning as discussed in Appendix D. Hence, for methodological study purposes, we believe that molecular docking provides a more useful and realistic test bed for algorithm development.
R ELATED W ORK
The REINVENT (Olivecrona et al., 2017) architecture consists of two recurrent neural network (RNN) architectures, generating molecules as tokenized SMILE strings.
The “Prior network” is trained with maximum likelihood estimation on a set of canonical SMILE strings, while the “Agent network” is trained with policy gradient and rewarded using a combination of task scores and
Prior network estimations.
The Junction Tree Variational Autoencoder ( JTVAE , Jin et al.
(2018)) trains two encoder/decoder networks in building a ﬁxed-dimension latent space representation of molecules, where one network captures junction tree structure of molecules and the other is responsible for ﬁne grain connectivity.
Novel molecules with desired properties are then generated using Bayesian optimization on the latent space.
Graph Convolutional Policy Network ( GCPN , You et al.
(2018a)) is a policy gradient RL architecture for de novo molecular generation.
The network deﬁnes domain-speciﬁc modiﬁcations on molecular graphs so that chemical validity is maintained at each episode.
Additionally, the model optimizes for realism with adversarial training and expert pre-training using trajectories generated from known molecules in the ZINC library.
Molecule
Deep Q-Networks ( MolDQN , Zhou et al.
(2019)) is a Q-learning model using Morgan ﬁngerprint as representations of molecules.
To achieve molecular validity, chemical modiﬁcations are directly deﬁned for each episode.
To enhance exploration of chemical space, MolDQN learns H independent
Q-functions, each of which is trained on separate sub-samples of the training data.
Markov Molecular Sampling ( MARS , Xie et al.
(2021)) generates molecules by employing an iterative method of editing fragments within a molecular graph, producing high-quality candidates through Markov chain Monte Carlo sampling (MCMC).
MARS then uses the MCMC samples in training a GNN to represent and select candidate edits, further improving sampling efﬁciency.
C ONCLUSIONS
In this work, we introduced a spatial graph attention mechanism and a curiosity-driven policy network to discover novel molecules optimized for targeted objectives.
We identiﬁed candidate antiviral compounds designed to inhibit the SARS-CoV-2 protein NSP15, leveraging extensive molecular docking simulations.
Our framework advances the state-of-the-art algorithms in the optimization of molecules with antiviral potential as measured by molecular docking scores, while maintaining reasonable synthetic accessibility.
We note that a valuable extension of our work would be to focus on lead-optimization — the reﬁnement of molecules already known to bind the protein of interest through position-constrained modiﬁcation.
Such knowledge-based and iterative reﬁnements may help to work around limitations of the accuracy of molecular docking predictions.
A CKNOWLEDGMENTS
This work was funded via the DOE Ofﬁce of Science through the National Virtual Biotechnology Laboratory (NVBL), a consortium of DOE national laboratories focused on the response to
COVID-19, with funding provided by the Coronavirus CARES Act.
This research used resources of the Oak Ridge Leadership Computing Facility (OLCF) at the Oak Ridge National Laboratory, which is supported by the Ofﬁce of Science of the U
S. Department of Energy under Contract No.
DE-AC05-00OR22725.
This manuscript has been coauthored by UT-Battelle, LLC under contract no.
DE-AC05-00OR22725 with the U.S. Department of Energy.
The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States
Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes.
The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-publicaccess-plan, last accessed September 16, 2020).
A PPENDIX A D ETAILED F ORMULATION OF THE P ROBLEM
Our goal is to establish a set of decision rules to generate graph-structured data that maximizes compound objectives under certain constraints.
Similar to prior formulations, the generating process is deﬁned as a time homogeneous Markov Decision Process (MDP).
We give a formal deﬁnition of this process in Appendix A.1.
Under this setting, the action policies and state transition dynamics at step t can be factorized according to the Markov property:
[Equation]
[Equation]
) where { s t a t } t are state-action sequences.
A reward function r ( s a ) is used to assess an action a taken at a given state s .
The process terminates at an optional stopping time T and s T is then proposed as the ﬁnal product of the current generating cycle.
We aim to estimate the optimal policy π in terms of various objectives to be constructed later in the experiment section.
A. M EASURE T HEORY C ONSTRUCTION OF M ARKOV D ECISION P ROCESS
Let ( S, S ) and ( A, A ) be two measurable spaces called the state space and action space; functions Π : S ×
A → R and T : S ×
A × S → R are said to be a policy and a transition probability respectively if
For each s ∈ S , E → Π( s, E ) is a probability measure on ( A, A ) ; for each ( s, a ) ∈ S × A , F → T ( s, a, F ) is a probability measure on ( S, S ) .
For each E ∈ A , s → Π( s, E ) is a measurable function from ( S, S )
→ ( R , B ) ; for each F ∈ S , ( s, a ) → T ( s, a, F ) is a measurable function from ( S × A, S ⊗ A )
→ ( R , B ) .
We say a sequence of random variable duples ( S t , A t ) deﬁned on the two measurable spaces is a
Markov decision chain if
[Equation]
A function r : S ×
A → R is said to be the reward function w.r.t.
the Markov decision chain if r ( s t , E t ) =
E Π ,T [ R ( s t +1 ) | S t = s t , A t ∈ E t ] where R : S → R is its underlying reward function.
With an abuse of notation, we deﬁne π ( a | s ) :
= Π( s, { a } ) , ρ ( s (cid:48) | s, a ) :
=
T ( s, a, { s (cid:48) } ) and let r ( s, a ) denote r ( s, { a } ) .
B L EARNING E NVIRONMENT AND R EWARD E VALUATION B. E NVIRONMENT - CR E M
Chemically Reasonable Mutations (CReM) is an open-source fragment-based framework for chemical structure modiﬁcation.
The use of libraries of chemical fragments allows for a direct control of the chemical validity of molecular substructures and to consider the chemical context of coupled fragments (e.g., resonance effects).
Compared to atom-based approaches, CReM explores less of chemical space but guarantees chemical validity for each modiﬁcation, because only fragments that are in the same chemical context are interchangeable.
Compared to reaction-based frameworks, CReM enables a larger exploration of chemical space but may explore chemical modiﬁcations that are less synthetically feasible.
Fragments are generated from the ChEMBL database (Gaulton et al., 2012) and for each fragment, the chemical context is encoded for several context radius sizes in a SMILES string and stored along with the fragment in a separate database.
For each query molecule, mutations are enumerated by matching the context of its fragments with those that are found in the CReM fragment-context database (Polishchuk, 2020).
In this work, we use grow function on a single carbon to generate initial choices if a warm-start dataset is not provided, and mutate function to enumerate possible modiﬁcations with the default context radius size of 3 to ﬁnd replacements.
B. E VALUATION - A UTO D OCK -GPU
Docking programs use the three-dimensional structure of the protein (i.e., the receptor) to predict the most stable bound conformations of the small molecules (i.e., its putative ligands) of interest, often targeting a pre-deﬁned functional site, such as the catalytic site.
An optimization algorithm within a scoring function is employed towards ﬁnding the ligand conformations that likely correspond to binding free energy minima.
The scoring function is conformation-dependent and typically comprises physics-based empirical or semi-empirical potentials that describe pair-wise atomic terms, such as dispersion, hydrogen bonding, electrostatics, and desolvation (Huang et al., 2010;
Huey et al., 2007).
AutoDock is a computational simulated docking program that uses a Lamarckian genetic algorithm to predict native-like conformations of protein-ligand complexes and a semiempirical scoring function to estimate the corresponding binding afﬁnities.
Lower values of docking scores indicate stronger predicted interactions.
The opposite value of the lowest estimated binding afﬁnity energy obtained for each molecule forms the reward.
AutoDock-GPU (Santos-Martins et al., 2021) is an extension of AutoDock to leverage the highlyparallel architecture of GPUs.
Within AutoDock-GPU, ADADELTA (Zeiler, 2012), a gradientbased method, is used for local reﬁnement.
The structural information of the receptor (here, the
NSP15 protein) used by AutoDock-GPU is processed prior to running the framework.
In this preparatory step, AutoDockTools (Morris et al., 2009b) was used to deﬁne the search space for docking on NSP15 (PDB ID 6W01; Figure 5) and to generate the PDBQT ﬁle of the receptor, which contains atomic coordinates, partial charges, and AutoDock atom types.
AutoGrid4 (Morris et al., 2009a) was used to pre-calculate grid maps of interaction energy at the binding site for the different atom types deﬁned in CReM.
In evaluation, after applying an initial ﬁlter within RDKit to check whether a given SMILES is chemically valid (i.e., hybridization, ring membership etc.), a 3D conformer of the molecule is generated using AllChem.
EmbedMolecule.
SMILES that do not correspond to valid compounds are discarded.
Next, the molecular geometry is energy minimized within RDKit using the generalized force ﬁled
MMFF94.
This also provides DGAPN/sGAT the “stable” 3D conformer of the compound, and is used as input for molecular docking via AutoDock-GPU.
We also excluded any molecules from the ﬁnal result set that were fully rigid and larger than the search box in the receptor.
This only occurred in two molecules from the JT-VAE evaluation.
C H YPERPARAMETER S ETTINGS FOR S INGLE - OBJECTIVE O PTIMIZATION
Based on a parameter sweep, we set number of GNN layers to be 3 , MLP layers to be 3 , withof the GNN layers and 0 of the MLP layers shared between query and key.
Number of layers in
RND is set to 1 ; all numbers of hidden neurons 256 ; learning rate for actor 5 − 4 , for critic 1 − 4 , for
RND 2 − 3 ; update time steps (i.e. batch size) 200 .
Number of epochs per iteration and clipping parameter (cid:15) for PPO are 30 and 0 .
Output dimensions and clipping parameter η for RND areand 5 .
In evaluation mode, we use arg max policy instead of sampling policy, expand the number of candidates per step from 15-20 to 128 and expand the maximum time steps per episode fromto 20 compared to training.
For more details regarding hyperparameter settings, see our codebase at https://github.com/yulun-rayn/DGAPN .
D M ORE R ESULTS ON QED AND P ENALIZED L OG P
Although QED and penalized LogP are the most popular objectives to benchmark ML algorithms for molecule generation, these benchmarks are questionable for both scientiﬁc study and practical use as Xie et al.
(2021) pointed out.
Most methods can obtain QED scores close or equal to the highest possible of 0.948, making the metric hard to distinguish different methods.
As for pLogP, if we simply construct a large molecule with no ring, such as the molecule from SMILES ‘CCCCC...CCCCC’ (139 carbons), it will give us a pLogP score of 50.31 which beats all state-of-the-art models in Table 4.
Needless to say, we will achieve a even higher pLogP by continuously adding carbons, which was exactly how REINVENT performed in our experiment.
We note that we were able to raise our results to around 18 solely by doubling the maximum time steps per episode reported in
Appendix C, yet not so interested in pushing the performance on this somewhat meaningless metric by continuously increasing one hyperparameter.
[Table]
The results from REINVENT were produced in our own experiments, while others were directly pulled out from the original results reported in the literature
. E D EFINITIONS OF QED AND SA E. Q UANTITATIVE E STIMATE OF D RUGLIKENESS
(QED) is deﬁned as
[Equation]
where d i are eight widely used molecular properties.
Speciﬁcally, they are molecular weight (MW), octanol-water partition coefﬁcient (ALOGP), number of hydrogen bond donors (HBD), number of hydrogen bond acceptors (HBA), molecular polar surface area (PSA), number of rotatable bonds (ROTB), the number of aromatic rings (AROM), and number of structural alerts.
For each d i , d i ( x ) =
[Equation]
i
[Equation]
each a i , . . .
, f i are given by a supplementary table in Bickerton et al.
E. S YNTHETIC A CCESSIBILITY
(SA) is deﬁned as
[Equation]
The fragment score is calculated as a sum of contributions from fragments of 934,046 PubChem already-synthesized chemicals.
The complexity penalty is computed from a combination of ringComplexityScore, stereoComplexityScore, macroCyclePenalty, and the sizePenalty: ringComplexityScore =
log( nRingBridgeAtoms + 1) + log( nSprioAtoms + 1) stereoComplexityScore
= log( nStereoCenters + 1) macroCyclePenalty
= log( nMacroCycles + 1)
sizePenalty =
nAtoms 1 .
005 − nAtoms