GRASP: Navigating Retrosynthetic Planning with Goal-driven Policy
Anonymous Author(s)
AffiliationAddress email
Abstract
Retrosynthetic planning occupies a crucial position in synthetic chemistry and,accordingly, drug discovery, which aims to find synthetic pathways of a targetmolecule through a sequential decision-making process on a set of feasible reactions.
While the majority of recent works focus on the prediction of feasiblereactions at each step, there have been limited attempts toward improving thesequential decision-making policy.
Existing strategies rely on either the expensiveand high-variance value estimation by online rollout, or a settled value estimationneural network pre-trained with simulated pathways of limited diversity and nonegative feedback.
Besides, how to return multiple candidate pathways that are notonly diverse but also desirable for chemists (e.g., affordable building block materials) remains an open challenge.
To this end, we propose a Goal-dRiven Actor-criticretroSynthetic Planning (GRASP) framework, where we identify the policy that performs goal-driven retrosynthesis navigation toward a user-demand objective.
Ourexperiments on the benchmark Pistachio dataset and a chemists-designed datasetdemonstrate that the framework outperforms existing state-of-the-art approachesby up to 32 .
2% on search efficiency and 5 .
6% on quality.
Remarkably, our userstudies show that GRASP successfully plans pathways that accomplish the goalprescribed with a goal (building block materials).
Introduction
Retrosynthetic planning has significantly advanced chemical synthesis, bringing in increasinglysophisticated medicines that cure diseases and materials that improve life.
A retrosynthetic plannertakes the structure of a target molecule as input and recursively selects feasible reactions to unsolvedintermediate molecules until eventually reaching building block molecules.
Since an unsolved intermediate molecule usually requires multiple steps of reactions to synthesize and at each step has upto hundreds of feasible reaction candidates, retrosynthetic planning with an enormous search spaceis very challenging even for experienced chemists.
Consequently, computer-aided synthesis planning (CASP) enters the scene to assist chemists in accelerating the process of designing retrosyntheticpathways.
Computer-aided retrosynthesis planning consists of: 1) a single-step retrosynthesis prediction whichpredicts a list of feasible reaction candidates that connect a target molecule to its respective precursors,and 2) a multi-step planning policy that searches for the optimal synthetic pathway by recursivelyapplying the single-step prediction model.
Recent years have witnessed a plethora of advancements insingle-step prediction models [26, 32, 20, 10], while in this work we are pursuing a more efficient andeffective planning policy that limits the effective search space to include the most likely successfulpathways.
Prior multi-step planning centered around tree or graph search methods [11, 22], where the searchis guided by only the total reaction cost (quality) from the target molecule to the current node.
To improve the search policy, recent attempts [25, 2, 7] include the estimated value from the current nodeto building block molecules, i.e., being building block aware.
Unfortunately, the value estimation in[25, 7] is by online roll-out, unfavorably being of high variance and low search efficiency.
Though[2] addressed this issue by pre-training a value network on simulated pathways, these pathwaysconstructed from an existing single-step reaction dataset offer limited diversity and no negativeexperiences to learn from.
Over and above, scoring and ranking the quality of many feasible pathways towards a target moleculehas been notoriously difficult.
The considerations that dictate a high-quality pathway, including highreaction yields, simple reaction conditions, and low building block molecule costs, are oftentimesconflicting and require a trade-off; moreover, predicting reaction yields [23] and conditions is verychallenging due to ill-defined and noisy annotations.
Chen et al.
[2] proposed to evaluate the qualitywith the negative log-likelihood of all reactions predicted by the single-step model, while it ispredicated on the assumption that frequent reactions are with high yields or easy conditions andbiased by the seen reactions that train the single-step model.
Keeping in mind that the objective ofretrosynthetic planning is to assist chemists, and in practice the challenge of quality evaluation canbe overcome by 1) returning as diverse feasible pathways as possible for chemists to weigh theirpreferences, and 2) returning the pathways that meet the qualifying conditions prescribed by chemists,e.g., a set of very cheap building block materials or easy-to-synthesize intermediate molecules.
Therefore, we are motivated to propose a Goal-dRiven Actor-critic retroSynthetic Planning (GRASP)framework.
Specifically, we formulate retrosynthesis planning as a reinforcement learning (RL)problem, where we first learn a policy network that takes continuous actions encoding the structurelevel molecular information to allow navigation in the huge discrete action space of single-stepreaction candidates.
Moreover, GRASP learns a goal-driven Q -value estimation network to updatethe policy, by sampling both successful (positive) and failed (negative) experiences and relabeling thegoals of sampled experiences.
Finally, the learned Q-value estimation and policy networks join toguide the Monte-Carlo Tree Search, after which GRASP returns diverse pathways as a result of agood exploration-exploitation tradeoff.
In summary, our contributions are threefold.•
We propose a novel actor-critic retrosynthetic planning framework GRASP, which learns fromextensive positive and negative experiences to navigate through huge single-step reaction spaces.•
We are the first to empower goal-driven planning, which mitigates the challenge in quality evaluationof pathways by directly fulfilling the requirements prescribed by chemists.• We have evaluated the performance of GRASP on both an academic and an industrial benchmarkdataset.
The results and user studies demonstrate that GRASP outperforms all baselines in generalretrosynthetic planning metrics by a significant margin and is the first to achieve high-qualitygoal-driven retrosynthetic planning.
Related Work
Single-step Retrosynthesis Prediction Single-step prediction models can be categorized into twomain classes, i.e., template-based and template-free.
Template-based methods rely on templatesthat encode chemical reaction cores to convert a product molecule into reactants.
The key isto rank templates and select an appropriate template to apply, for which recent attempts [3, 24]solve the problem of template selection through a classification neural network.
Despite theirsuperior interpretability, template-based approaches are disadvantaged by 1) the daunting challengeof atom-mapping for template extraction, and 2) poor generalization to unknown reaction typesor structures beyond templates.
On the other hand, template-free methods [14, 13, 21], inspiredby the recent progress of seq2seq [28] and Transformer [29], regard single-step retrosynthesisprediction as a translation task and translate a product molecule represented in SMILES strings [30]to reactant SMILES strings.
To join the benefits of template-based and template-free methods, recentworks [26, 32, 20] seek semi-template-based methods where the reaction center dictating a reactionis firstly predicted via graph neural networks and the resulting intermediate synthons are secondlytranslated into reactants via seq2seq or graph translation models.
Recently, Kim et al.
[10] proposed tofine-tune a single-step prediction model with the feedback from a multi-step retrosynthetic planningpolicy, leading to a search-guided single-step model.
We have validated in Section 4.3 that ourplanning policy is also compatible with the framework and improves single-step prediction andthereby final pathways.
Multi-step Retrosynthetic planning We summarize the comparison of existing multi-step retrosynthetic planning policies in Table 1.
Specifically, previous planning methods HgSearch [22] and the
Neural-guidedsearch building blocks awareness
Negativeexperiences Exploration-exploitationtradeoff
Goal-driven
HgSearch [22] ✗ ✓ ✗ ✓ ✗
DFPN-E [11] ✗ ✗ ✗ ✗ ✗
MCTS [25] ✗ ✓ ✓ ✓ ✗
Retro* [2] ✓ ✓ ✗ ✗ ✗
Ours ✓ ✓ ✓ ✓ ✓ proof number search [11] are traditional heuristic search algorithms, where the chemical feasibilityand the value of the negative (failure) pathways are not considered.
Inspired by AlphaGo [27], Segleret al.
[25] adopted the Monte-Carlo tree search to generate a search tree on the fly and explore andgenerate multiple synthesis pathways.
Unfortunately, each node with a combination of all precursorsin a reaction leads to enormous search space, and the value estimation by vanilla online roll-out is ofhigh variance and high computation cost.
Inspired by [11], Chen et al.
[2] designed a neural-based A*-like algorithm that learns an additional value network with automatically constructed andonly successful pathways to bias the search prior.
Recently, Han et al.
[9] and Xie et al.
[31] useda GNN-based value network to capture inter-molecular/intra-pathway level information to furtherimprove the A*-like retrosynthetic planning algorithm.
However, one disadvantage of A*-like [2]retrosynthetic planning algorithm is that it fails to balance exploration and exploitation, resultingin less diverse pathways.
Moreover, none of the previous approaches is capable of biasing theretrosynthetic planning toward a favorable goal prescribed by chemists.
Methods
First, Sec. 3.1 introduces Markov decision process (MDP) setting for goal-driven retrosyntheticplanning.
Secondly, Sec. 3.2 elaborates on the framework of the GRASP goal-driven actor-criticagent and the training procedure with TD3 [6] algorithm.
Lastly, Sec. 3.3 introduces GRASP planningfor a given target molecule under a goal-driven variant of MCTS.
3.1 Goal-driven MDP for retrosynthetic planning
We denote a finite-horizon MDP by M = {S , A , T , G , r ( s, a, g ) , H, γ } for our goal-driven retrosynthetic planning task.
We use s ∈ S to denote the state (molecule) space, a ∈ A to denote theaction (reaction) space which consists of reaction candidates a generated by the single-step predictionmodel, and T ( s t +1 | s t , a t ) to denote the state transition from s t to s t +1 through performing reactiona with a deterministic state transition probability.
We denote the goal space as G , which has the samesize as the state space S since our goal is to navigate toward particular states.
Considering the ultimategoal for retrosynthesis is to discover retrosynthetic pathways reaching building block molecules, wedenote the goal for the entire set of building block molecules as G B , where each goal g i ∈ G B indicatesa specific building block molecule i .
To simultaneously adapt to both general (non-goal-driven) andgoal-driven retrosynthetic planning, we define g =
G B as all zero embedding and concatenate thegoal embedding with an additional binary feature embedding, where we use I ( g = G B ) = 0 for thegeneral planning and I ( g = g i )
=
for the goal-driven planning towards goal g i .
For the rewarddesign of r ( s, a, g ) , we assign the goal-driven path-finding reward as r ( s, a, g ) =
when the state sreaches the desired goal g after taking action a and r ( s, a, g ) = 0 otherwise.
Finally, γ is the discountfactor, and H is the maximum horizon (length) for the pathway.
3.2 GRASP framework and training procedure
GRASP has two parameterized components as shown in Fig.
1: actor network denote by π ϕ ( a | s, g )and critic network by Q
θ ( s, a, g ) .
In the setting of retrosynthetic planning, we regard the upstreamsingle-step retrosynthesis predictor as the environment and retrosynthetic planner as the agent .
At each time step t , the agent outputs a proto-action ˜ a t with the same size as action embedding,according to its goal-driven policy network π ϕ ( ˜ a t | s t , g ) from observing the current state s t and goalg .
Since we are unaware of the possible goal state for a given initial state without prior knowledge,we use g = G B in π ϕ ( ˜ a t | s t , g = G B ) as behavioral policy.
Specifically, we add a small amount of random noises N to the action for exploration during sampling:
[Equation]
After acquiring the proto-action ˜ a t ′ , the agent has to identify an actual reaction a t from availablereaction candidates A ( s t ) given the state s t and action embedding ˜ a t .
Inspired by the k-nearestneighbor (k-NN) trick for large discrete action space similar to the Wolpertinger training [5], weuse the true action embeddings from the available actions
A ( s t ) for the k-NN calculation duringthe action selection procedure.
Furthermore, we may encounter reactions that induce more thanone non-building block molecule as reactants, namely convergent synthesis.
Convergent synthesisreaction, although infrequent in retrosynthesis , introduces a variation in the cardinality of staterepresentation s that conflicts with MDP settings.
For previous work in MCTS for retrosynthesis, Segler et al. [25] accumulates all non-building block reactants as a set of molecules in staterepresentation, but only performs action selection on a single molecule.
The combinatorial nature ofstate representation introduces bias in reward propagation and sparsity in variance estimation.
Toovercome this complexity during the sampling and training phase, we use the average distance amongall reactants for k-NN computation each time we encounter a reaction with convergent synthesis.
Asa result, we obtain an actual reaction a t by referring to the k-NN computation of the proto-action ˜ a tover the available actions A ( s t ) .
If a convergent synthesis reaction is identified as the true action a tby the environment, non-building block reactants are split into separate next states as independenttrajectories to perform parallel sampling.
Eventually, the next state s t +1 is defined as the non-buildingblock molecule among the reactants of a t .
The sampling of a trajectory terminates when the statereaches the goal g or the length of the trajectory reaches the maximum horizon H .
Goal-driven relabeling: To capture the goal-driven planning insights from a retrosynthesis pathwayand accelerate learning in the sparse reward setting, we are inspired by [1] to relabel transition tuplesin trajectories.
The core idea of applying goal-driven relabeling in retrosynthesis is to exploit thedata generated from the general retrosynthesis policy π ϕ ( a | s, g = G B ) to train featured retrosynthesisplanning data, and incorporate the agent with knowledge of navigating toward a specific goal stateg =
g i .
In practice, we copy the state transition tuple M i =
( s i , a i , r i ( .
| g i =
G B ) , s i +1 ) andrandomly relabel the tuple M i
= ( s i , a i , r i ( .
| g ′ i ) , s i +1 ) with a relabeling probability p r using futurerelabeling strategy.
Specifically, for the i th tuple M ⟩ in trajectory τ with length T , we performgoal-driven relabeling by iterating over all future transitions as:
[Equation]
for k ∈ (0 , T − i ] .
Since the relabeling probability p r is an important hyperparameter to balancebetween general and goal-driven planning, we will further examine the effect of different p r on theplanning performance in the experiment.
The RL agent is trained with TD3 [6] algorithm.
For tuple i in a training batch, the target criticnetwork is first updated using the one-step TD equation as:
[Equation]
where, Q ′ and π ′ denote the target critic and actor networks with fixed parameters copied fromoriginal critic and actor networks Q θ and π ϕ respectively, and r i , s i +1 , g i +1 represents the reward,state, and goal at the step t .
With the TD target y i , we can calculate the batch mean-square-error losson the original critic network Q θ ( s, a, g ) as:
[Equation]
Since the goal of the actor network is designed to maximize the overall return (success rate), and thegoal of critic network is to approximate the overall return, the actor π ϕ can be trained by maximizingthe Q value by minimizing:
[Equation]
Self-imitation learning: To learn from highly imbalanced pathways in the overall search space( > 85% failures), we adopt self-imitation learning [18] (SIL) to accelerate the convergence in sparse
[Figure]
reward and enhance the training efficiency.
Intuitively, SIL assists the RL agent to emphasize highquality planning experiences.
Instead of using the Bellman equation for calculating the target Q-value,
SIL directly uses the Monte-Carlo return of each ‘good’ episode as the Q-value target.
It is crucialfor the agent to exploit success trajectories in retrosynthetic planning tasks, especially during theearly stage of training when a large proportion of samples in the replay buffer originated from failedtrajectories.
We denote the simplified SIL loss for i th tuple in a success trajectory τ with length h as:
[Equation]
where y sili =
(cid:80) hi =
k
γ h − k r i .
We also include the full training algorithm as Alg. 1 in Appendix B.
3.3 GRASP retrosynthetic planning
In this section, we demonstrate the GRASP planning procedure for a target molecule and a specificgoal with GRASP RL agent π ϕ ( a | s, g ) and Q θ ( a, s, g ) .
Since each newly expanded molecule node is the same as the initial state in GRASP , it is natural tocombine our RL agent into Monte-Carlo tree search (MCTS) with goal-driven p-UCT function [27]:
[Equation]
In previous MCTS for retrosynthesis , Segler et al. [25] used an online roll-out stage for MonteCarlo estimation of success rate for each leaf node, which both suffer from high-variance and heavycomputation.
Therefore, one of the key differences between GRASP and Segler et al.
during theplanning stage is we completely skip the online roll-out stage and directly refer to the RL agent forvalue estimation of the leaf nodes instead of Monte-Carlo estimation from the online roll-out.
Toalign with our MDP settings in RL, we adopt a goal-driven MCTS planning with individual moleculesas tree node representation.
Specifically, our framework consists of three phases as shown in Fig. 2,and for simplicity, we ignore all building block molecules in the figure since no selection action willbe performed on:
• Selection:
Starting from the root node, the p-UCT function in Eq.
5 is used to iteratively selectan action.
At any step t , available actions in candidate set a t ∈ A ( s t ) and respective single-stepconfidence score p c ( a t | s t ) are provided by the single-step retrosynthesis predictor, and we define:
[Equation]
where D ( · ) is the same distance metric used in the k-NN calculation, ˜ a is produced by policynetwork π θ (˜ a | s t , g ) , and N ( s t − 1 , a t − 1 ) denotes the visit count of the state-action pair of previous
[Figure]
B
C
E
D
F
[Figure]
E
D
F
Q * Q *
Q * Q *
A
B
C
E
D
F
Q * Q *
Q * Q * Q *
Q * Q *
Q * Q *
[Figure] [Figure]
[Figure] [Figure]
states.
If a convergent synthesis action with multiple non-building block reactants is selected, we perform parallel selection and select all non-building block reactants as the next state.
We iterativelyperform selection on states until reaching a leaf node.
Eventually, a set of leaf nodes is identified forexpansion.
• Expansion: Each leaf node s t from the selected set is expanded by referring to the single-step model.
Each available action from A ( s t ) is directly appended to the node s t .
For convergent synthesisaction, we generate the same number of leaf nodes depending on the quantity of unsolved molecules.
For each newly generated leaf node, we evaluate their Q ∗ value with the following rule: if there isno available action for s t , we directly apply Q ∗ = 0 .
Before applying Q θ ( s, a, g ) network to assign
Q ∗ value for newly added leaf nodes, we assign Q ∗ = 1 and label it as ‘solved’ if the s t reachesg or Q ∗ =
if the state reaches the maximum horizon.
If the state is undetermined, we assign
Q ∗ =
Q θ ( s t +1 , a, g ) by applying Q θ value network.
• Update:
During the update phase, the Q ∗ values and visit counts N ( s, a ) are traversed backward following the
selection path from leaf nodes back to the root node.
We use a simple moving averagefor updating Q value with a discount factor γ :
[Equation]
Experiments
4.1 Experiment setup
4.1.1 Baseline Algorithms and Evaluation metrics
We compare our approach against a varieties of baselines including: MCTS [25], DFPN-E [11],
Retro*-0 and Retro* [2], and Hyper-Graph Search (HgSearch) [22].
MCTS[25]: The original version of MCTS for retrosynthesis planning which exploits online roll-outto obtain Monte-Carlo estimation of future value without data generalization.
DFPN-E[11]: Depth-first proof number search (PNS) performed on AND-OR tree with an additivereaction likelihood as cost.
Retro* and Retro*-0 [2]: Different from DPFN-E, Retro* utilizes the AND-OR tree as a singleplayer game and utilizes global value estimation.
Additionally, Retro* pre-trains a value network ona simulated retrosynthesis pathway dataset.
Retro*-0 denotes its version that performs the searchwithout the value network.
Retro* is reported as the state-of-the-art search algorithm for retrosyntheticplanning.
HgSearch[22]: HgSearch is a beam-search-like algorithm performed on a hyper-graph structure.
Theheuristics are the product of the single-step confidence score and molecular complexity score (SCScore) [4].
We use four different metrics to comprehensively evaluate the performance of different searchalgorithms: 1.
Pathway length: We use the total number of reactions in the retrosynthesis pathwayfor length evaluation.
Pathway cost: The cost function is defined as the summation of the negativelog-likelihood (confidence score) of the reactions in the pathway τ provided by the single-step model, i.e., − (cid:80) a ∈ τ log p c ( a | s ) [2].
The cost is also regarded as a criterion for chemical feasibility.3.
Planning efficiency: Since the primary objective of AI-aided retrosynthesis is to help chemistsfind successful pathways faster, efficiency has been a crucial evaluation criterion for a multi-stepretrosynthesis planning algorithm.
Therefore, we follow [2] to take the number of single-step inferencecalls as a qualified surrogate of time , as single-step inference ( ∼ 2s per iter) takes up almost > 99%of the time (only ∼ 0.006s per iter on planning).
Success rate: With a fixed number of single-stepinference calls, the success rate is defined as the percentage of solved molecules in the entire set.
4.1.2 Single-step retrosynthesis predictor
We adopt the template-free single-step retrosynthesis predictor based on molecular transformer (MT)[21, 13] from Schwaller et al.
[22] as our single-step retrosynthesis predictor.
Specifically,
Schwaller et al.
separately trained a pair of backward single-step generation models and forwardsingle-step prediction models, and cooperatively utilized them to generate high-quality single-stepretrosynthetic candidates with a confidence score p c ( a | s ) ranging from 0 to 1.
Both statistics [14, 13]and our user study in real-world scenarios demonstrate that MT-based single-step framework achieveshigher accuracy and less chemo-selectivity when compared with template-based approaches.
Eventually, we choose top- k =100 reactions ranked according to the confidence score predicted by thesingle-step model as the available single-step candidate set for a given molecule since top- k =100 issufficient to represent feasible single-step reaction space for a molecule.
4.2 Creating benchmark datasets
Single-step reactions and building block molecules dataset: We use the Pistachio reactiondataset (Ver. 18.11.19)
[16] as our benchmark dataset for training our single-step models, andthe implementation details are listed in Appendix A.1.
After further pruning and discarding reactionswith multiple products, the entire dataset consists of 2.7M reactions.
The dataset is further splitrandomly into train/val/test sets following 90%/5%/5% proportions.
We use the complete 231Mcommercially available molecules presented in eMolecules 1 for the building block molecule set.
Pathway dataset: Since only Retro* requires an additional simulated retrosynthesis pathway datasetfor pre-training its value network for planning, we follow the setting in [2] and construct the Pistachiopathway dataset similarly.
Specifically, we obtained 61554 pathways with an average length of 3.66.
We split the dataset into 40000 training pathways, 21354 validation pathways, and 200 test pathwaysfor Retro* value network training.
Note that constructing an artificial pathway dataset by simplyconcatenating single-step reactions is only a reference rather than an optimal/expert pathway fora given molecule in a given search space.
Moreover, an expert pathway dataset is unavailable forunreported molecules and expensive to obtain in real-world scenarios.
Therefore for a fair comparison,the target molecules in the pathway dataset are simultaneously used as initial states for GRASPtraining.
Expert dataset: We also include a real-world expert dataset ‘WuxiTest’ designed by WuxiAppTecchemists, and each target molecule is provided with one reference pathway.
WuxiTest consists of 500molecules that are specifically designed to consist only of molecules that have never appeared in anyjournals and patents.
Molecules were split into ten categories in terms of retrosynthesis strategies, andeach category shares similar molecular substructures.
We partition the pathway dataset category-wiseas 80%/10%/10% into train/valid/test sets as partitions and follow the same training settings as the
Pistachio.
4.3 Results
The performance of all methods is listed in Table.
For both the Pistachio and WuxiTest datasets, ourapproach achieves the highest success rate compared with the baselines.
We observe that HgSearchachieves the best performance on the average cost metric in Pistachio, mainly from the near-exhaustivesearch performed on less challenging molecules.
Our approach outperforms other baselines in averageexpansion by a large margin, demonstrating the performance gain in planning efficiency broughtby RL training.
In the WuxiTest dataset, our approach outperforms all other baselines in all fourmetrics.
Since the WuxiTest dataset is designed to emphasize retrosynthesis strategies with morechallenging but strategically similar molecules, the result proves that RL training can generalizeplanning knowledge from a molecule with similar substructures.
We demonstrate the influenceof time limit on the success rate for different approaches for the WuxiTest dataset in Fig.
3a for
N max = 400 .
We also demonstrate that the success rate tends to saturate when for N > 400 byextending to N max = 1000 in Appendix C for all approaches. [Table]
Goal-driven planning performance Since GRASP is the first and only approach that empowersgoal-driven planning, to evaluate whether GRASP is capable of generating high-quality goal-drivenresults
, we conduct a double-blind user study of goal-driven planning on the WuxiTest.
Specifically,we run GRASP using the building block molecules in the source pathways from different baselinesas the GRASP’s goal input to obtain a goal-driven result.
In addition, we also include goal-drivenplanning using the general (GRASP’s general planning without specifying goal) and expert (goal inthe reference routes from the chemists) source pathways.
The results in Table.
3 demonstrate that ourapproach can perform goal-oriented search and in the meantime generate a high-quality result.
Weprovide an exemplar of pathway comparison in reference for demonstration in Fig.
5 and Fig. 6 in Appendix C.
[Table]
Compatibility to self-improved retrosynthetic planning Self-improved retrosynthetic planning [10]is an end-to-end framework that fine-tunes the single-step model to imitate successful trajectoriesfound by a fixed search (Retro* was used in the original work) algorithm by altering the priordistribution of single-step candidates in the search space.
To evaluate the adaptation of GRASPwith the self-improved framework, we follow the training procedure in [10] by replacing Retro*with GRASP and observe the performance on Pistachio and WuxiTest.
As shown in Fig.3b, theself-improved retrosynthetic planning framework can improve the success rate of GRASP by usingits own planning experience to fine-tune the single-step model.
4.4 Ablation studies
In this section, we investigate the following questions from different ablation studies: 1.
The influenceof two components: goal-driven relabeling (GDR) and self-imitation learning (SIL) over episodicreward during RL training.
How does the different probability of GDR affect the performance ofgeneral retrosynthesis planning and goal-driven retrosynthesis planning?
Influence of different components: We cross-check the training statistics of four different combinations: GRASP without GDR and SIL, GRASP with GDR, GRASP with SIL, and GRASP with
GDR and SIL.
We evaluate the results on Pistachio by calculating the average reward with respect totraining episodes.
In binary reward setting, we use success rate as the criteria for reward evaluation,and the result is shown in Fig.3c.
On the one hand, SIL significantly improves the overall trainingstatistics but induces a more significant variance in the training process.
The variance is attributedto higher variance when using Monte-Carlo return and inevitable trade-offs in gradient propagationfrom different successful pathways under the same target.
On the other hand, GDR also offers a [Figure] certain amount of performance enhancement by relieving training difficulties induced by the sparserewards.
However, we are more interested in GDR’s contribution brought to goal-driven planning.
Influence of GDR probability:
The main hyper-parameter we are interested in is GDR probability,which adjusts the distribution of transition tuples in the replay buffer for general planning and goaldriven planning.
Specifically, we use the WuxiTest to evaluate the trade-off between the success rateof general and goal-driven planning with respect to GDR probability.
For goal-driven planning, weuse the building block molecules in expert pathways for goal-driven input.
The result is shown in
Fig.3d.
We observe that the success rate of goal-driven planning is lower than general planning asexpected, as it requires both general success and the specific goal reached.
However, the successrate for goal-driven planning improves significantly when relabeling probability ranges from 10% to70% .
Nevertheless, high relabeling probability impairs the success rate for general planning since
GDR might incur failures in general planning pathways, and increasing the proportion of goal-drivendata leads to less proportion of general data.
In conclusion, it is crucial to select an appropriate GDRprobability depending on the actual usage of GRASP .
Conclusion
This paper proposes GRASP, a novel goal-driven retrosynthetic planning approach.
Unlike existingapproaches that limit their generalization planning knowledge in a static dataset, GRASP can capturesynthetic knowledge through self-generated experiences.
Moreover, GRASP can perform goaldriven retrosynthetic planning that none of the existing approaches could explicitly accomplish.
Experimental results on academic and industrial benchmark datasets demonstrate GRASP outperformsall baselines in general retrosynthetic planning and first achieves high-quality goal-driven planning.
Future works could extend to designing a more informative goal space representation with additionalretrosynthetic planning level knowledge, possibly through pre-training an encoder for retrosyntheticpathways [17].
Although we have successfully deployed the proposed GRASP in a real-worldpharmaceutical company to expedite the retrosynthetic planning process, the planning results from
GRASP still need to be monitored or directed by personnel with chemical expertise if the targetmolecule is a novel or a rarely reported compound.
The community is lacking thorough researchon the robustness of different single-step/multi-step retrosynthetic planning algorithms and how togeneralize to out-of-distribution molecules, which we think could be an appealing research direction.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] We will publish
the code upon acceptance.
Datasets will be provided upon valid license.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e
) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]