GRASP: Navigating Retrosynthetic Planning with Goal-driven Policy
Abstract
Retrosynthetic planning occupies a crucial position in synthetic chemistry and, accordingly, drug discovery, which aims to find synthetic pathways of a target molecule through a sequential decision-making process on a set of feasible reactions.
While the majority of recent works focus on the prediction of feasible reactions at each step, there have been limited attempts toward improving the sequential decision-making policy.
Existing strategies rely on either the expensive and high-variance value estimation by online rollout, or a settled value estimation neural network pre-trained with simulated pathways of limited diversity and no negative feedback.
Besides, how to return multiple candidate pathways that are not only diverse but also desirable for chemists (e.g., affordable building block materials) remains an open challenge.
To this end, we propose a Goal-dRiven Actor-critic retroSynthetic Planning (GRASP) framework, where we identify the policy that performs goal-driven retrosynthesis navigation toward a user-demand objective.
Our experiments on the benchmark Pistachio dataset and a chemists-designed dataset demonstrate that the framework outperforms existing state-of-the-art approaches by up to 32 .
2% on search efficiency and 5 .
6% on quality.
Remarkably, our user studies show that GRASP successfully plans pathways that accomplish the goal prescribed with a goal (building block materials).
Introduction
Retrosynthetic planning has significantly advanced chemical synthesis, bringing in increasingly sophisticated medicines that cure diseases and materials that improve life.
A retrosynthetic planner takes the structure of a target molecule as input and recursively selects feasible reactions to unsolved intermediate molecules until eventually reaching building block molecules.
Since an unsolved intermediate molecule usually requires multiple steps of reactions to synthesize and at each step has up to hundreds of feasible reaction candidates, retrosynthetic planning with an enormous search space is very challenging even for experienced chemists.
Consequently, computer-aided synthesis planning (CASP) enters the scene to assist chemists in accelerating the process of designing retrosynthetic pathways.
Computer-aided retrosynthesis planning consists of: 1) a single-step retrosynthesis prediction which predicts a list of feasible reaction candidates that connect a target molecule to its respective precursors, and 2) a multi-step planning policy that searches for the optimal synthetic pathway by recursively applying the single-step prediction model.
Recent years have witnessed a plethora of advancements in single-step prediction models [25, 31, 19, 10], while in this work we are pursuing a more efficient and effective planning policy that limits the effective search space to include the most likely successful pathways.
Prior multi-step planning centered around tree or graph search methods [11, 21], where the search is guided by only the total reaction cost (quality) from the target molecule to the current node.
To improve the search policy, recent attempts [24, 2, 7] include the estimated value from the current node to building block molecules, i.e., being building block aware.
Unfortunately, the value estimation in [24, 7] is by online roll-out, unfavorably being of high variance and low search efficiency.
Though [2] addressed this issue by pre-training a value network on simulated pathways, these pathways constructed from an existing single-step reaction dataset offer limited diversity and no negative experiences to learn from.
Over and above, scoring and ranking the quality of many feasible pathways towards a target molecule has been notoriously difficult.
The considerations that dictate a high-quality pathway, including high reaction yields, simple reaction conditions, and low building block molecule costs, are oftentimes conflicting and require a trade-off; moreover, predicting reaction yields [22] and conditions is very challenging due to ill-defined and noisy annotations.
Chen et al.
[2] proposed to evaluate the quality with the negative log-likelihood of all reactions predicted by the single-step model, while it is predicated on the assumption that frequent reactions are with high yields or easy conditions and biased by the seen reactions that train the single-step model.
Keeping in mind that the objective of retrosynthetic planning is to assist chemists, and in practice the challenge of quality evaluation can be overcome by 1) returning as diverse feasible pathways as possible for chemists to weigh their preferences, and 2) returning the pathways that meet the qualifying conditions prescribed by chemists, e.g., a set of very cheap building block materials or easy-to-synthesize intermediate molecules.
Therefore, we are motivated to propose a Goal-dRiven Actor-critic retroSynthetic Planning (GRASP) framework.
Specifically, we formulate retrosynthesis planning as a reinforcement learning (RL) problem, where we first learn a policy network that takes continuous actions encoding the structurelevel molecular information to allow navigation in the huge discrete action space of single-step reaction candidates.
Moreover, GRASP
learns a goal-driven Q -value estimation network to update the policy, by sampling both successful (positive) and failed (negative) experiences and relabeling the goals of sampled experiences.
Finally, the learned Q-value estimation and policy networks join to guide the Monte-Carlo Tree Search, after which GRASP returns diverse pathways as a result of a good exploration-exploitation tradeoff.
In summary, our contributions are threefold.
• We propose a novel actor-critic retrosynthetic planning framework GRASP, which learns from extensive positive and negative experiences to navigate through huge single-step reaction spaces.
• We are the first to empower goal-driven planning, which mitigates the challenge in quality evaluation of pathways by directly fulfilling the requirements prescribed by chemists. • We have evaluated the performance of GRASP on both an academic and an industrial benchmark dataset.
The results and user studies demonstrate that GRASP outperforms all baselines in general retrosynthetic planning metrics by a significant margin and is the first to achieve high-quality goal-driven retrosynthetic planning.
Related Work
Single-step Retrosynthesis Prediction Single-step prediction models can be categorized into two main classes, i.e., template-based and template-free.
Template-based methods rely on templates that encode chemical reaction cores to convert a product molecule into reactants.
The key is to rank templates and select an appropriate template to apply, for which recent attempts [3, 23] solve the problem of template selection through a classification neural network.
Despite their superior interpretability, template-based approaches are disadvantaged by 1) the daunting challenge of atom-mapping for template extraction, and 2) poor generalization to unknown reaction types or structures beyond templates.
On the other hand, template-free methods [14, 13, 20], inspired by the recent progress of seq2seq [27] and Transformer [28], regard single-step retrosynthesis prediction as a translation task and translate a product molecule represented in SMILES strings [29] to reactant SMILES strings.
To join the benefits of template-based and template-free methods, recent works [25, 31, 19] seek semi-template-based methods where the reaction center dictating a reaction is firstly predicted via graph neural networks and the resulting intermediate synthons are secondly translated into reactants via seq2seq or graph translation models.
Recently, Kim et al.
[10] proposed to fine-tune a single-step prediction model with the feedback from a multi-step retrosynthetic planning policy, leading to a search-guided single-step model.
We have validated in Section 4.3 that our planning policy is also compatible with the framework and improves single-step prediction and thereby final pathways.
[Table]
Multi-step Retrosynthetic planning We summarize the comparison of existing multi-step retrosynthetic planning policies in Table 1.
Specifically, previous planning methods HgSearch [21] and the proof number search [11] are traditional heuristic search algorithms, where the chemical feasibility and the value of the negative (failure) pathways are not considered.
Inspired by AlphaGo [26], Segler et al.
[24] adopted the Monte-Carlo tree search to generate a search tree on the fly and explore and generate multiple synthesis pathways.
Unfortunately, each node with a combination of all precursors in a reaction leads to enormous search space, and the value estimation by vanilla online roll-out is of high variance and high computation cost.
Inspired by [11], Chen et al.
[2] designed a neural-based
A*-like algorithm that learns an additional value network with automatically constructed and only successful pathways to bias the search prior.
Recently, Han et al.
[9] and Xie et al.
[30] used a GNN-based value network to capture inter-molecular/intra-pathway level information to further improve the A*-like retrosynthetic planning algorithm.
However, one disadvantage of A*-like [2] retrosynthetic planning algorithm is that it fails to balance exploration and exploitation, resulting in less diverse pathways.
Moreover, none of the previous approaches is capable of biasing the retrosynthetic planning toward a favorable goal prescribed by chemists.
Methods
First, Sec. 3.1 introduces Markov decision process (MDP) setting for goal-driven retrosynthetic planning.
Secondly, Sec. 3.2 elaborates on the framework of the GRASP goal-driven actor-critic agent and the training procedure with TD3 [6] algorithm.
Lastly, Sec. 3.3 introduces GRASP planning for a given target molecule under a goal-driven variant of MCTS.
3.1 Goal-driven MDP for retrosynthetic planning
We denote a finite-horizon MDP by M = {S , A , T , G , r ( s, a, g ) , H, γ } for our goal-driven retrosynthetic planning task.
We use s ∈ S to denote the state (molecule) space, a ∈ A to denote the action (reaction) space which consists of reaction candidates a generated by the single-step prediction model, and T ( s t +1 | s t , a t ) to denote the state transition from s t to s t +1 through performing reaction a with a deterministic state transition probability.
We denote the goal space as G , which has the same size as the state space S since our goal is to navigate toward particular states.
Considering the ultimate goal for retrosynthesis is to discover retrosynthetic pathways reaching building block molecules, we denote the goal for the entire set of building block molecules as G B , where each goal g i ∈ G B indicates a specific building block molecule i .
To simultaneously adapt to both general (non-goal-driven) and goal-driven retrosynthetic planning, we define g =
G B as all zero embedding and concatenate the goal embedding with an additional binary feature embedding, where we use I ( g = G B ) = {+
+} for the general planning and I ( g = g i )
=
for the goal-driven planning towards goal g i .
For the reward design of r ( s, a, g ) , we assign the goal-driven path-finding reward as r ( s, a, g ) =
when the state s reaches the desired goal g after taking action a and r ( s, a, g ) = 0 otherwise.
Finally, γ is the discount factor, and H is the maximum horizon (length) for the pathway.
3.2 GRASP framework and training procedure
GRASP has two parameterized components as shown in Fig.
1: actor network denote by π ϕ ( a | s, g ) and critic network by Q
θ ( s, a, g ) .
In the setting of retrosynthetic planning, we regard the upstream single-step retrosynthesis predictor as the environment and retrosynthetic planner as the agent .
At each time step t , the agent outputs a proto-action ˜ a t with the same size as action embedding, according to its goal-driven policy network π ϕ ( ˜ a t | s t , g ) from observing the current state s t and goal g .
Since we are unaware of the possible goal state for a given initial state without prior knowledge, we use g = G B in π ϕ ( ˜ a t | s t , g = G B ) as behavioral policy.
Specifically, we add a small amount of random noises N to the action for exploration during sampling:
[Equation]
After acquiring the proto-action ˜ a t ′ , the agent has to identify an actual reaction a t from available reaction candidates A ( s t ) given the state s t and action embedding ˜ a t .
Inspired by the k-nearest neighbor (k-NN) trick for large discrete action space similar to the Wolpertinger training [5], we use the true action embeddings from the available actions A ( s t ) for the k-NN calculation during the action selection procedure.
Furthermore, we may encounter reactions that induce more than one non-building block molecule as reactants, namely convergent synthesis.
Convergent synthesis reaction, although infrequent in retrosynthesis , introduces a variation in the cardinality of state representation s that conflicts with MDP settings.
In previous work in MCTS for retrosynthesis,
Segler et al.
[24] accumulates all non-building block reactants as a set of molecules in state representation, but only performs action selection on a single molecule.
The combinatorial nature of state representation introduces bias in reward propagation and sparsity in variance estimation.
To overcome this complexity during the sampling and training phase, we use the average distance among all reactants for k-NN computation each time we encounter a reaction with convergent synthesis.
As a result, we obtain an actual reaction a t by referring to the k-NN computation of the proto-action ˜ a t over the available actions A ( s t ) .
If a convergent synthesis reaction is identified as the true action a t by the environment, non-building block reactants are split into separate next states as independent trajectories to perform parallel sampling.
Eventually, the next state s t +1 is defined as the non-building block molecule among the reactants of a t .
The sampling of a trajectory terminates when the state reaches the goal g or the length of the trajectory reaches the maximum horizon H .
Goal-driven relabeling: To capture the goal-driven planning insights from a retrosynthesis pathway and accelerate learning in the sparse reward setting, we are inspired by [1] to relabel transition tuples in trajectories.
The core idea of applying goal-driven relabeling in retrosynthesis is to exploit the data generated from the general retrosynthesis policy π ϕ ( a | s, g = G B ) to train featured retrosynthesis planning data, and incorporate the agent with knowledge of navigating toward a specific goal state g
= g i .
In practice, we copy the state transition tuple M i =
( s i , a i , r i ( .
| g i =
G B ) , s i +1 ) and randomly relabel the tuple M i =
( s i , a i , r i ( .
| g ′ i ) , s i +1 ) with a relabeling probability p r using future relabeling strategy.
Specifically, for the i th tuple M ⟩ in trajectory τ with length T , we perform goal-driven relabeling by iterating over all future transitions as:
[Equation]
for k ∈ (0 , T − i ] .
Since the relabeling probability p r is an important hyperparameter to balance between general and goal-driven planning, we will further examine the effect of different p r on the planning performance in the experiment.
The RL agent is trained with TD3 [6] algorithm.
For tuple i in a training batch, the target critic network is first updated using the one-step TD equation as:
[Equation]
where, Q ′ and π ′ denote the target critic and actor networks with fixed parameters copied from original critic and actor networks Q θ and π ϕ respectively, and r i , s i +1 , g i +1 represents the reward, state, and goal at the step t .
With the TD target y i , we can calculate the batch mean-square-error loss on the original critic network Q θ ( s, a, g ) as:
[Equation]
[Figure] Since the goal of the actor network is designed to maximize the overall return (success rate), and the goal of critic network is to approximate the overall return [Figure] the actor π ϕ can be trained by maximizing the Q value by minimizing:
[Equation]
Self-imitation learning: To learn from highly imbalanced pathways in the overall search space ( > 85% failures), we adopt self-imitation learning [17] (SIL) to accelerate the convergence in sparse reward and enhance the training efficiency.
Intuitively, SIL assists the RL agent to emphasize highquality planning experiences.
Instead of using the Bellman equation for calculating the target Q-value,
SIL directly uses the Monte-Carlo return of each ‘good’ episode as the Q-value target.
It is crucial for the agent to exploit success trajectories in retrosynthetic planning tasks, especially during the early stage of training when a large proportion of samples in the replay buffer originated from failed trajectories.
We denote the simplified SIL loss for i th tuple in a success trajectory τ with length h as:
[Equation]
where y sili =
(cid:80) hi =
k
γ h − k r i .
We also include the full training algorithm in Alg.
3.3 GRASP retrosynthetic planning
In this section, we demonstrate the GRASP planning procedure for a target molecule and a specific goal with GRASP RL agent π ϕ ( a | s, g ) and Q θ ( a, s, g ) .
Since each newly expanded molecule node is the same as the initial state in GRASP , it is natural to combine our RL agent into Monte-Carlo tree search (MCTS) with goal-driven p-UCT function [26]:
[Equation]
In previous MCTS for retrosynthesis, Segler et al.
[24] used an online roll-out stage for MonteCarlo estimation of success rate for each leaf node, which both suffer from high-variance and heavy computation.
Therefore, one of the key differences between GRASP and Segler et al.
during the planning stage is we completely skip the online roll-out stage and directly refer to the RL agent for value estimation of the leaf nodes instead of Monte-Carlo estimation from the online roll-out.
To align with our MDP settings in RL, we adopt a goal-driven MCTS planning with individual molecules Initialize Critic network Q θ , Q ′ θ and actor network π ϕ , π ′ ϕ , replay buffer B , initial state space S , action space A , goal space G , reward function r : S × A × G → R . for ep = 1 to M do
Sample initial state s 0 ∈ S . for t = 0 to H do Sample proto-action ˜ a t using behavioral policy and exploration noise for general planning π ′ ϕ ( a | s t , G B ) Perform k-NN computation and execute action a t
Observe reward r t = r
( s t , a t , G B ) and next state s t + end for for t = 0 to H do
Store original transition ( s t , a t , r t , s t +1 , g ) in B
Generate transition copy and relabel g ′
=
s t
+ i and r ′ t = r
( s t , a t , g ′ ) with probability p r with future goal-driven relabeling strategy.
Store transition ( s t , a t , r ′ t , s t +1 , g ′ ) in B end for for t = 0 to N do
Perform actor-critic batched TD training on θ, ϕ with Eq.2 and Eq.
Perform SIL training on θ with Eq.
end for end for as tree node representation.
Specifically, our framework consists of three phases as shown in Fig.
2, and for simplicity, we ignore all building block molecules in the figure since no selection action will be performed on:
• Selection:
Starting from the root node, the p-UCT function in Eq.
5 is used to iteratively select an action.
At any step t , available actions in candidate set a t ∈ A ( s t ) and respective single-step confidence score p c ( a t | s t ) are provided by the single-step retrosynthesis predictor and we define:
[Equation]
where D ( · ) is the same distance metric used in the k-NN calculation, ˜ a is produced by policy network π θ (˜ a | s t , g ) , and N ( s t − 1 , a t − 1 ) denotes the visit count of the state-action pair of previous states.
If a convergent synthesis action with multiple non-building block reactants is selected, we perform parallel selection and select all non-building block reactants as the next state.
We iteratively perform selection on states until reaching a leaf node.
Eventually, a set of leaf nodes is identified for expansion.
• Expansion: Each leaf node s t from the selected set is expanded by referring to the single-step model.
Each available action from A ( s t ) is directly appended to the node s t .
For convergent synthesis action, we generate the same number of leaf nodes depending on the quantity of unsolved molecules.
For each newly generated leaf node, we evaluate their Q ∗ value with the following rule: if there is no available action for s t , we directly apply Q ∗ = 0 .
Before applying Q θ ( s, a, g ) network to assign Q ∗ value for newly added leaf nodes, we assign Q ∗ = 1 and label it as ‘solved’ if the s t reaches g or Q ∗ =
if the state reaches the maximum horizon.
If the state is undetermined, we assign Q ∗ =
Q θ ( s t +1 , a, g ) by applying Q θ value network.
• Update:
During the update phase, the Q ∗ values and visit counts N ( s, a ) are traversed backward following the selection path from leaf nodes back to the root node.
We use a simple moving average for updating Q value with a discount factor γ :
[Equation]
A
B
C
E D F A
B
C
E
D
F
Q *
Q *
Q *
Q *
A
B
C
E
D
F
Q *
Q *
Q *
Q *
Q *
Q * Q *
Q *
Q *
Q *
Q * Q *
Q * (1) Selection
Iteratively apply selection policy (p- UCT) until reaching leaf nodes (labeled red).
(2) Expansion
Expand selected leaf nodes and generate new leaf nodes (green) and predict value with Q * Q ( s , a , g ) (3) Update
Update upward along the selected edges until reaching
[Figure]
Experiments 4.1 Experiment setup
Baseline Algorithms: We compare our approach against a varieties of baselines including: 1.
MCTS[24]: The original vanilla version of MCTS for retrosynthesis planning which exploits online roll-out to obtain Monte-Carlo estimation of future value without data generalization.
DFPN-E[11]:
Depth-first proof number search (PNS) performed on AND-OR tree with an additive reaction likelihood as cost.
Retro* and Retro*-0 [2]: Different from DPFN-E, Retro* utilizes the AND-OR tree as a single-player game and utilizes global value estimation.
Additionally, Retro* pre-trains a value network on a simulated retrosynthesis pathway dataset.
Retro*-0 denotes its version that performs the search without the value network.
Retro* is reported as the state-of-the-art search algorithm for retrosynthetic planning.
Hyper-Graph Search (HgSearch)[21]: HgSearch is a beam-search-like algorithm performed on a hyper-graph structure.
The heuristics are the product of the single-step confidence score and molecular complexity score (SCScore) [4].
Evaluation metrics: We use four different metrics to comprehensively evaluate the performance of different search algorithms: 1.
Pathway length: We use the total number of reactions in the retrosynthesis pathway for length evaluation.
Pathway cost: The cost function is defined as the summation of the negative log-likelihood (confidence score) of the reactions in the pathway τ provided by the single-step model, i.e., − (cid:80) a ∈ τ log p c ( a | s ) [2].
The cost is also regarded as a criterion for chemical feasibility.
Planning efficiency: Since the primary objective of AI-aided retrosynthesis is to help chemists find successful pathways faster, efficiency has been a crucial evaluation criterion for a multi-step retrosynthesis planning algorithm.
Therefore, we follow [2] to take the number of single-step inference calls as a qualified surrogate of time , as single-step inference ( ∼ 2s per iter) takes up almost > 99% of the time (only ∼ 0.006s per iter on planning).
Success rate: With a fixed number of single-step inference calls, the success rate is defined as the percentage of solved molecules in the entire set.
Single-step retrosynthesis predictor: We adopt the template-free single-step retrosynthesis predictor based on molecular transformer (MT)[20, 13] from Schwaller et al.
[21] as our single-step retrosynthesis predictor.
Specifically, Schwaller et al.
separately trained a pair of backward single-step generation models and forward single-step prediction models, and cooperatively utilized them to generate high-quality single-step retrosynthetic candidates with a confidence score p c ( a | s ) ranging from 0 to 1.
Both statistics [14, 13] and our user study in real-world scenarios demonstrate that
MT-based single-step framework achieves higher accuracy and less chemo-selectivity when compared with template-based approaches.
Eventually, we choose top- k =100 reactions ranked according to the confidence score predicted by the single-step model as the available single-step candidate set for a given molecule since top- k =100 is sufficient to represent feasible single-step reaction space for a molecule. [Table]
4.2 Creating benchmark datasets
Single-step reactions and building block molecules dataset: We use the Pistachio reaction dataset (Ver. 18.11.19)
[16] as our benchmark dataset for training our single-step models, and the implementation details are listed in Appendix A.1.
After further pruning and discarding reactions with multiple products, the entire dataset consists of 2.7M reactions.
The dataset is further split randomly into train/val/test sets following 90%/5%/5% proportions.
We use the complete 231M commercially available molecules presented in eMolecules 1 for the building block molecule set.
Pathway dataset: Since only Retro* requires an additional simulated retrosynthesis pathway dataset for pre-training its value network for planning, we follow the setting in [2] and construct the Pistachio pathway dataset similarly.
Specifically, we obtained 61554 pathways with an average length of 3.66.
We split the dataset into 40000 training pathways, 21354 validation pathways, and 200 test pathways for Retro* value network training.
Note that constructing an artificial pathway dataset by simply concatenating single-step reactions is only a reference rather than an optimal/expert pathway for a given molecule in a given search space.
Moreover, an expert pathway dataset is unavailable for unreported molecules and expensive to obtain in real-world scenarios.
Therefore for a fair comparison, the target molecules in the pathway dataset are simultaneously used as initial states for GRASP training.
Expert dataset: We also include a real-world expert dataset ‘WuxiTest’ designed by WuxiAppTec chemists, and each target molecule is provided with one reference pathway.
WuxiTest consists ofmolecules that are specifically designed to consist only of molecules that have never appeared in any journals and patents.
Molecules were split into ten categories in terms of retrosynthesis strategies, and each category shares similar molecular substructures.
We partition the pathway dataset category-wise as 80%/10%/10% into train/valid/test sets as partitions and follow the same training settings as the
Pistachio.
4.3 Results
The performance of all methods is listed in Table.
For both the Pistachio and WuxiTest datasets, our approach achieves the highest success rate compared with the baselines.
We observe that HgSearch achieves the best performance on the average cost metric in Pistachio, mainly from the near-exhaustive search performed on less challenging molecules.
Our approach outperforms other baselines in average expansion by a large margin, demonstrating the performance gain in planning efficiency brought by RL training.
In the WuxiTest dataset, our approach outperforms all other baselines in all four metrics.
Since the WuxiTest dataset is designed to emphasize retrosynthesis strategies with more challenging but strategically similar molecules, the result proves that RL training can generalize planning knowledge from a molecule with similar substructures.
We demonstrate the influence of time limit on the success rate for different approaches for the WuxiTest dataset in Fig.
3a for
N max = 400 .
We also demonstrate that the success rate tends to saturate when for N > 400 by extending to N max = 1000 in Appendix C for all approaches. W UXI T EST
R ETRO * H G S EARCH DFPN-E MCTS GRASPG ENERAL GRASPE XPERT S OURCE A VG . L ENGTH 7. N/A GRASP A VG .
L ENGTH 7.
S OURCE A VG .
R ATING (0-10) 7.
N/A
[Table]
VG
[Table]
Goal-driven planning performance Since GRASP is the first and only approach that empowers goal-driven planning, to evaluate whether GRASP is capable of generating high-quality goal-driven results
, we conduct a double-blind user study of goal-driven planning on the WuxiTest.
Specifically, we run GRASP using the building block molecules in the source pathways from different baselines as the GRASP’s goal input to obtain a goal-driven result.
In addition, we also include goal-driven planning using the general (GRASP’s general planning without specifying goal) and expert (goal in the reference routes from the chemists) source pathways.
The results in Table.
3 demonstrate that our approach can perform goal-oriented search and in the meantime generate a high-quality result.
We provide an exemplar of pathway comparison in reference for demonstration in Fig.
5 and Fig. 6 in
Appendix C.
Compatibility to self-improved retrosynthetic planning Self-improved retrosynthetic planning [10] is an end-to-end framework that fine-tunes the single-step model to imitate successful trajectories found by a fixed search (Retro* was used in the original work) algorithm by altering the prior distribution of single-step candidates in the search space.
To evaluate the adaptation of GRASP with the self-improved framework, we follow the training procedure in [10] by replacing Retro* with GRASP and observe the performance on Pistachio and WuxiTest.
As shown in Fig.3b, the self-improved retrosynthetic planning framework can improve the success rate of GRASP by using its own planning experience to fine-tune the single-step model.
4.4 Ablation studies
In this section, we investigate the following questions from different ablation studies: 1.
The influence of two components: goal-driven relabeling (GDR) and self-imitation learning (SIL) over episodic reward during RL training.
How does the different probability of GDR affect the performance of general retrosynthesis planning and goal-driven retrosynthesis planning?
Influence of different components: We cross-check the training statistics of four combinations: GRASP without GDR and SIL, GRASP with GDR, GRASP with SIL, and GRASP with GDR and
SIL.
We evaluate the results on Pistachio by calculating the average reward with respect to training episodes.
In binary reward setting, we use success rate as the criteria for reward evaluation, and the result is shown in Fig.3c.
On the one hand, SIL significantly improves the overall training statistics but induces a more significant variance in the training process.
The phenomenon is attributed to higher variance when using Monte-Carlo return and inevitable trade-offs in gradient propagation from different successful pathways under the same target.
On the other hand, GDR also offers a certain amount of performance enhancement by relieving training difficulties induced by the sparse rewards.
However, we are more interested in GDR’s contribution to goal-driven planning.
Influence of GDR probability: The main hyperparameter we are interested in is GDR probability, which adjusts the distribution of transition tuples in the replay buffer for general and goal-driven planning.
Specifically, we use the WuxiTest to evaluate the trade-off between general and goal-driven planning success rate for the hyperparameter controlling GDR probability.
For goal-driven planning, we use the building block molecules in expert pathways for goal-driven input.
The result is shown in
Fig.3d.
As expected, we observe that the success rate of goal-driven planning is lower than general planning, as it requires both general success and the specific goal reached.
However, the success rate for goal-driven planning improves significantly when relabeling probability ranges from 10% to 70% .
Nevertheless, high relabeling probability impairs the success rate for general planning since
GDR might incur failures in general planning pathways, and increasing the proportion of goal-driven data leads to less proportion of general data.
In conclusion, it is crucial to select an appropriate GDR probability depending on the actual usage of GRASP . [Figure] u cce [Figure]
Conclusion
This paper proposes GRASP, a novel goal-driven retrosynthetic planning approach.
Unlike existing approaches that limit their generalization planning knowledge in a static dataset, GRASP can capture synthetic knowledge through self-generated experiences.
Moreover, GRASP can perform goaldriven retrosynthetic planning that none of the existing approaches could explicitly accomplish.
Experimental results on academic and industrial benchmark datasets demonstrate GRASP outperforms all baselines in general retrosynthetic planning and first achieves high-quality goal-driven planning.
We have deployed GRASP in a real-world company to expedite the planning of synthetic pathways toward novel chemical compounds.
By transforming the originally expertise-intensive pathway discovery process into efficient automation, GRASP significantly reduces the cost of the workforce on pathway design in both medicinal and process chemistry.
More importantly, GRASP supports the high customization of goals and further alleviates the cost of human labor on pathway screening and post-processing.
We believe that our work will significantly inspire related research on more efficient and chemists-machine interactive retrosynthetic planning frameworks.
Nevertheless, the planning results from GRASP still need to be monitored or directed by personnel with chemical expertise if the target molecule is a novel or a rarely reported compound.
The community lacks thorough research on the robustness of different single-step/multi-step retrosynthetic planning algorithms and how to generalize to out-of-distribution molecules, which we think could be an appealing research direction.
Acknowledgement
This work is sponsored by the Starry Night Science Fund at Shanghai Institute for Advanced Study (Zhejiang University) and Program of Zhejiang Province Science and Technology (2022C01044).
A Experimental settings A.1 Single-step retrosynthesis model training details
We use the Pistachio dataset to train our single-step retrosynthesis model and SMILES to represent molecules. Pistachio contains 6.9M reaction details. The majority are extracted from experimental procedure text in patents, including 4.2M reactions in the United States Patent Office (USPTO). The remaining 1.8M is extracted from sketches in U.S. patents.
Results from previous work [21] have demonstrated that the single-step model trained by the Pistachio dataset achieves higher chemical class diversity and disconnection options than the model trained with the USPTO dataset.
We follow the setting in [21] by training a forward reaction prediction model and a backward retrosynthesis model for the single-step model, using the molecular transformer [20] architecture.
Specifically, both transformers are implemented using OpenNMT-py [12], as encoder and decoder, with hyperparameters listed in Tab. 4.
The training takes about 50 hours on a single NVIDIA RTX 3090 with caching.
[Table]
We utilize Morgan fingerprint bit vector with dimension 1024 and radius 2 in RDKit as molecule representation in states, actions, and goals, which contains similar information as Extended-connectivity
Fingerprints (ECFP4) [18].
Chemical fingerprints have long been used to represent molecules, including the classic Morgan fingerprints that utilize the graph topology to include neighbour information.
Recently, numerous deep learning algorithms directly use Morgan fingerprint for molecular representation as input for neural networks in retrosynthesis tasks [8, 23].
Furthermore, works [15] have demonstrated that ECFP4 fingerprints have similar performance compared to state-of-the-art graph-based deep learning algorithms. We utilize Morgan fingerprint bit vector with dimension 1024 and radius 2 in RDKit 2 , which contains similar information as Extended-connectivity Fingerprints (ECFP4) [18].
We listed all hyper-parameters in Tab. 5 for GRASP network hyper-parameters and training hyper-parameters.
The training takes about 95 hours for 500k episodes on Pistachio and 20 hours for 50k episodes on WuxiTest on a single NVIDIA RTX 3090.
To accelerate training, we implemented a cache system to store the single-step reaction candidates for encountering repeated molecules during training phase since the main computational cost is inferencing the single-step MT model.
[Table]
Performance saturation: We demonstrate the performance (success rate and avg.
time) saturation of success rate with respect to the number of single-step inference calls N max = 1000 in Fig. GRASP still outperforms the baselines, though all algorithms tend to converge as stated [24].
Exemplar figures: We present an exemplar comparison between a source pathway in Fig.
5 and a goal-driven pathway Fig.
6 for the user study conducted in goal-driven experiments, where we use the red molecule as goal input for GRASP planning.
We also present an exemplar in Fig.
7 of the
GRASP planning tree for reference.