Mind the Retrosynthesis Gap: Bridging the divide between Single-step and Multi-step Retrosynthesis Prediction
Anonymous Author(s)
AffiliationAddress
Abstract
Retrosynthesis is the task of breaking down a chemical compound recursively stepby-step into molecular precursors until a set of commercially available moleculesis found.
Consequently, the goal is to provide a valid synthesis route for a molecule.
As more single-step models develop, we see increasing accuracy in the prediction of molecular disconnections, potentially improving the creation of syntheticpaths.
Multi-step approaches repeatedly apply the chemical information storedin single-step retrosynthesis models.
However, this connection is not reflectedin contemporary research, fixing either the single-step model or the multi-stepalgorithm in the process.
In this work, we establish a bridge between both tasks bybenchmarking the performance and transfer of different single-step retrosynthesismodels to the multi-step domain by leveraging two common search algorithms,
Monte Carlo Tree Search and Retro*.
We show that models designed for single-stepretrosynthesis, when extended to multi-step, can have a tremendous impact on theroute finding capabilities of current multi-step methods, improving performance byup to +30% compared to the most widely used model.
Furthermore, we observeno clear link between contemporary single-step and multi-step evaluation metrics,showing that single-step models need to be developed and tested for the multi-stepdomain and not as an isolated task to find synthesis routes for molecules of interest.
Introduction
Artificial intelligence has greatly impacted the drug discovery pipeline by achieving human-likeperformance in the field of retrosynthesis [1].
Retrosynthesis is the task of breaking down a chemicalcompound recursively into molecular precursors until a set of commercially available buildingblock molecules is found [1, 2].
Consequently, the goal is to provide a valid synthesis route for amolecule.
Potential applications of these synthesis routes are suggestions for medical chemists onhow to produce a molecule of interest [3], a foundation for autonomous chemistry [4] and usingsynthesizability as part of De Novo Drug Design [5].
The field of computational retrosynthesis prediction is separated into two different tasks [6].
Insingle-step retrosynthesis, the goal is to find the likely precursors or reactants for a given product.
Inmulti-step retrosynthesis planning, the goal is to find viable synthesis paths over multiple reactionsteps.
The task of single-step retrosynthesis prediction is treated as a supervised learning task, commonlycategorized as template-based or template-free [7].
Template-based approaches use manually curatedor data-driven reaction templates [8].
These templates represent the general atoms and bond structuresrequired to perform a reaction.
Therefore the objective is to predict the most appropriate templateto break down the molecule [9, 10].
Template-free approaches are treated as a sequence predictionproblem predicting one token of a chemical SMILES vocabulary at a time [11, 12], drawing inspirationfrom natural language processing [13].
Recently variations of these two approaches have emerged.
In semi-template-based [14, 15], a molecule is first broken down into subparts, and then the subpartsare rebuilt into chemically viable reactants.
Lastly, though many models leverage the sequence-based
SMILES notation, there are also attempts to utilize graph-based descriptors across these approaches,exploiting the advantages of a molecular graph [10, 16].
In comparison to single-step retrosynthesis, multi-step retrosynthesis planning focuses on researchingnovel route search algorithms using a fixed single-step model to identify retrosynthetic disconnections.
The pioneering work in the field uses neural-guided Monte-Carlo Tree Search (MCTS) and a templatebased approach to find synthesis routes [1].
Instead of assessing the state value in the search tree atrun-time, alternative methods use oracle functions to guide the tree search.
These methods include
Depth-First Proof-Number (DFPN) search with edge cost, which combines classical DFPN witha neural heuristic [17], and Retro*, which combines the A* path finding algorithm with a neuralheuristic [18].
Newer approaches use a template-free model, either combining neural-guided MCTSwith reaction feasibility heuristics [19] or directly using synthesizability heuristics combined with aforward synthesis model [20].
Instead of using heuristics, self-play [21], learning a value function byletting an algorithm play the game of synthesis against itself, is an additional investigated approach[22, 23, 24].
Multi-step approaches repeatedly apply the chemical information stored in single-step retrosynthesismodels.
However, the relationship between single-step models and multi-step approaches is notreflected in contemporary research, treating both tasks as distinct entities.
Even though multistep algorithms require the use of single-step models, these single-step models are generally fixed.
Similarly, single-step models are developed without evaluating their use in multi-step approaches.
Therefore, an open question is how single-step retrosynthesis evaluation metrics translate to themulti-step domain [6] and, consequently, how single-step models affect the synthesis route findingcapabilities as part of a multi-step algorithm.
In this work, we establish a bridge between single andmulti-step retrosynthesis tasks by benchmarking the performance and transfer of different single-stepretrosynthesis models to the multi-step domain.
We show the impressive impact of the single-stepmodel on multi-step performance but, more importantly, a disconnection between contemporarysingle-step and multi-step evaluation metrics.
Methods
We select three state-of-the-art retrosynthesis single-step models to compare their performance inthe multi-step domain.
The model selection is based on dominant contemporary neural networkapproaches, i.e., contrastive learning, sequence-to-sequence, and graph-based encoding, consideringtheir respective top-1 to top-50 performance on the USPTO-50k single-step retrosynthesis benchmark[25, 26].
Accordingly, the selected models are MHNreact [9], a contrastive learning approach,
Chemformer [12], a sequence-to-sequence approach, and LocalRetro [10], a graph-based approach.
As an additional baseline, a template-based multi-layer perceptron approach [27, 28], drawinginspiration from [1], is included since it is often used in multi-step retrosynthesis algorithms [1, 17,18, 24].
Given that we aim to evaluate the capacity of these single-step retrosynthesis models inmulti-step retrosynthesis planning, we use the model hyperparameters suggested in their respectivepublications (Appendix A1) assuming the models are optimized for the single-step prediction task.
The only exception is Chemformer, where we use beam size 50 to produce the single-step results andbeam size 10, the publication default, for multi-step retrosynthesis planning.
To ensure the correct implementation of the single-step models and compare their single-step performance, we perform a 10-fold cross-validation by splitting the data into 80% training, 10% validation,and 10% test splits for each fold.
Each model is trained using the train split, training is monitoredusing the validation split, and the test split is used for final evaluation.
All models use the same datasplit for each fold, and the data is preprocessed according to the specifications of each model.
Each single-step model is evaluated by measuring its accuracy and inference time.
Single-stepaccuracy [7] is defined as the percentage of target compounds for which the model finds the groundtruth reactants within the top-k, k ∈ { 1 , 3 , 5 , 10 , 5 } , measuring the ability of the model to capturechemical reaction information.
Inference time is defined as the time needed to produce retrosynthesispredictions for a set of molecules, measuring the ability of the model to provide predictions in atimely manner.
In an ablation study, we measure the impact of the amount of evaluation data andbatch size on the inference time.
For the first, we measure the influence of doubling the evaluationdata while using the default batch size (Appendix A2), analyzing the scalability of the model.
For thesecond, we measure the impact of setting the batch size to 1, replicating the necessary conditions fora multi-step search algorithm that can only explore one molecule per instance (e.g. [1]).
The selected multi-step algorithms to evaluate the performance of the different single-step models are
MCTS [1], which dynamically assesses the state values of the search tree at run-time, and Retro*[18], which instead uses an oracle function.
We defer from using a self-play algorithm since it wouldbe necessary to retrain the self-play algorithm per problem instance, i.e., the set of used buildingblocks.
In the first and second experiments, the search settings for MCTS and Retro* are set to a time limit of1800 seconds (30 minutes) and 200 algorithm iterations per molecule (Appendix A3), respectively.
In a third experiment, the search settings for Retro* are set to a time limit of 28800 seconds (8 hours)to allow the single-step models to reach the maximum iteration limit (Retro*-extended)
(Appendix
A3), given their potential slow inference times.
This third experiment is only conducted with Retro*because the algorithm does not need to do multiple single-step model calls to evaluate a tree-searchstate.
Thus the algorithm is more likely to allow the single-step model to reach the iteration limit.
In this case, a single-step model call refers to the suggestion of multiple candidate reactants given aproduct.
In all cases, we search up to a maximum route length, or tree depth, of 7 and use the Zincstock of 17,422,831 building blocks [28].
All experiments are conducted by extending the opensource AiZynthFinder (AZF) multi-step retrosynthesis framework [28] to use alternative single-stepmodels instead of the thus far implemented baseline template-based model.
To evaluate the performance of a single-step model within a multi-step setting, we measure thesolvability of molecules when searching for a synthesis plan.
Solvability is the percentage of testmolecules for which a specific combination of search algorithm and single-step model can producesolved routes.
A route is considered solved when all predicted leaf compounds are available withinthe building block stock.
To further investigate the performance of single-step retrosynthesis models,we also analyze the average number of iterations carried out by the search algorithm, the averagenumber of calls to the single-step model, and the average search time, calculated across the testcompounds.
The data used for all experiments is USPTO-50k [25, 26], a commonly used dataset within the singlestep retrosynthesis field.
The dataset consists of 50,016 unique products and their respective reactants,where the randomly split dataset contains 40,012 training reactions, 5,002 validation reactions, and5,002 test reactions.
The multi-step evaluation is conducted on the products of the test set.
Single-step retrosynthesis models are trained and benchmarked on one Tesla V100 32GB GPU.
Incomparison, multi-step retrosynthesis experiments are evaluated using a high-performance CPU
[Table]
cluster to facilitate the parallelization necessary to evaluate an extensive set of molecules in anappropriate time frame.
Results
3.1 Single-step retrosynthesis prediction
We reproduce the performance of the selected single-step models using a 10-fold cross-validationwith USPTO-50k.
By averaging across the folds, we reproduce the results reported in Chemformer[12], MHNreact [9], and LocalRetro [10] (Fig. 1 and Appendix A4).
For all models the data splithas no discernable effect on the accuracy, shown by the small standard deviation across all folds.
Additionally, we calculate the performance of the baseline model implemented in AZF [28] as abenchmark to compare the single-step models.
For top-1 accuracy, Chemformer outperforms the othermodels, with an average accuracy of 54.7% (± 1.1%).
LocalRetro and MHNreact follow this with52.5% (± 0.7%) and 49.8% (± 0.8%) accuracy, respectively, and AZF performs notably worse withan accuracy of 43.3% (± 1.0%).
This pattern, however, is not maintained across the top-k measures.
Accuracy noticeably ascends within the top-3 for all models, LocalRetro seeing a +24.1% increasein accuracy to 76.6% (± 0.6%), similar to MHNreact with a +23.0% increase to 72.8% (± 1.0%).
AZF has a +16.8% increase in accuracy to 60.0% (± 1.0%), and Chemformer has the smallest gain inaccuracy with an +11.2% increase to 65.9% (± 1.0%).
Within the top-50 predictions, LocalRetroshows 96.6% (± 0.3%) accuracy, followed by MHNreact with 93.3% (± 0.4%) accuracy, both showingsimilar profiles across the top-k. AZF notably increases its performance across top-3 to top-10, giving78.1% (± 0.7%) accuracy at top-50.
Surprisingly, Chemformer delivers the lowest accuracy in thetop-50 of the models tested, with an accuracy of 73.3% (± 0.3%).
Though Chemformer outperformsother models in top-1, it is less able to find the ground-truth reactants for the remaining productsdespite additional explored alternatives with higher top-k.
The influence of increased data and decreased batch size on the single-step model inference timeis examined since single-step models typically evaluate in batches.
Generally, there is a linearrelationship when doubling the amount of inferred data (Fig. 2).
We observe that the inference timeper molecule remains stable when the amount of test data increases, except for LocalRetro, whichtriples the inference time per molecule.
In contrast, by decreasing batch size to one, we emulate theconditions of the model call within multi-step retrosynthesis planning.
Chemformer and MHNreactboth substantially increase their average inference time per molecule.
For Chemformer, the increasein inference time is discernible, reaching eight times compared to the default batch size.
MHNreacthas the most marked increase in inference time, with the change from batch size 32 to 1 leading to 18x longer inference time per molecule.
On the other hand, the inference time per molecule of
LocalRetro is hardly affected by this change.
3.2 Multi-step retrosynthesis planning
Introducing the single-step models into the selected search algorithms, MCTS generally performsworse than Retro* (Table 1).
In detail, Retro* performs better in terms of solvability, number ofexplored routes, and solved routes per molecule across nearly all tested single-step models.
The onlyexception is Chemformer, which produces more solved routes per molecule (MCTS: 3.33, Retro*:2.06) while using fewer model calls (MCTS: 8.48, Retro*: 14.4).
However, Chemformer with MCTSstill has a lower overall solvability (MCTS: 44.3%, Retro*: 53.4%).
In essence, it produces multiplesolved routes for a smaller subset of solved molecules.
Retro*-extended reaches or improves theresult of Retro*, given that single-step models have more time for inference.
In detail, Chemformerand MHNreact achieve higher performance using Retro*-extended, leveraging more single-stepmodel calls.
In comparison, the baseline AZF model and LocalRetro do not utilize the added timewith more single-step model calls as they already reach the 200 iteration limit within the 30 min timelimit of Retro*, thus having similar performance using both settings.
For the overall best-performing search setting, Retro*-extended, the single-step model ranking interms of solvability is LocalRetro (80.6%), Chemformer (65.6%), MHNreact (60.9%), and AZF(50.6%).
However, high solvability does not always imply a high number of solved routes.
Forexample, Chemformer has a higher solvability than MHNreact, yet produces a considerably lowernumber of solved routes per molecule (Chemformer: 8.04, MHNreact: 56.6).
Moreover, a highnumber of explored routes is also not directly connected to a high solvability.
For example, MHNreactexplores the highest number of routes per molecule but performs only the third best in solvabilitysince it solves comparably few explored routes.
Lastly, there are large disparities across the average search time per molecule.
Chemformer is by farthe slowest model (18737 sec), followed by MHNreact (8016 sec), both of which are considerablyslower compared to LocalRetro (322 sec) and AZF (129 sec).
Even with these extensive search times,
Chemformer and MHNreact do not reach the same level of single-step model calls as LocalRetro and
AZF.
Generally, LocalRetro outperforms other models in terms of solvability and number of solved routeswhile producing slightly fewer total explored routes than MHNreact and needing approximately 2.5xthe time per molecule in comparison to the fastest AZF baseline.
[Table]
Discussion
We show that a single-step retrosynthesis model can tremendously impact multi-step retrosynthesisplanning, influencing the ability to solve products and successfully produce multiple solutions.
Across all three experiments (MCTS, Retro*, Retro*-extended), the alternative single-step modelsmostly outperform the baseline AZF model.
In MCTS, one single-step model, LocalRetro, showsa considerably higher solvability than AZF.
In the case of Retro* and Retro*-extended, all modelsoutperform AZF, particularly when given an extended time to carry out sufficient model calls.
Thegenerally best performing model is LocalRetro, which has outstanding solvability and provides themost solved routes per molecule across all multi-step retrosynthesis planning experiments, continuallyoutperforming all other methods.
We show that the exchange of the single-step model alone canimprove solvability by +30.0%, reaching 80.6%, and triple the number of solved routes per moleculeto 151.
As such, the single-step model should be well considered when developing multi-stepretrosynthesis planning approaches.
Given our results, no clear pattern supports the usage of single-step top-k metrics as a potential proxymeasure for solvability in multi-step retrosynthesis.
For single-step models, the accuracy rankingvaries from top-1 to top-50 (Fig. 1 and Tab. A4).
However, these rankings, and their intermediates,are never matched by their respective multi-step solvability rankings (Tab. 1).
Hence, multi-stepsolvability does not solely depend on a singular single-step factor and should not be reduced to asingular top-k single-step metric.
Exclusively focusing on the top-1 accuracy of single-step models is especially problematic whentransferring the model to a multi-step domain.
Though Chemformer shows the highest top-1 accuracy,in multi-step experiments, it finds a comparatively low number of solved routes per molecule despitehaving the second highest solvability.
A cause for this could be its single-step accuracy profile, goingfrom the best performing to the worst performing model as top-k increases.
This suggests that themodel is proficient at predicting certain reactions but cannot find a diverse set of solutions.
However,diverse solutions are beneficial for a tree-search setting, where up to 50 possible explorable routealternatives could be added per search iteration.
Importantly, similar top-k accuracy profiles do not result in the same multi-step results.
For example,
MHNreact and LocalRetro have a similar top-k accuracy profile in single-step retrosynthesis butdiffer greatly in multi-step retrosynthesis planning.
Though they explore a similar number of routes,
MHNreact solves considerably fewer routes.
This difference in performance may be explainedby comparing Retro* and Retro*-extended, where MHNreact improves only slightly in solvabilitydespite having considerably more time.
Though MHNreact explores and solves many more routes ingeneral, the difference in solvability shows that it can only do this for molecules it had already solvedin the shorter time frame.
Since multi-step solvability is not solely dependent on the top-k accuracy shown by the single-stepmodels, other factors may contribute to their performance.
For example, given that search algorithmsgenerally have a limited run-time, single-step model inference times can greatly affect performancein multi-step retrosynthesis planning.
To produce solved routes, single-step models must carry out asmany single-step model calls as possible within an allocated time limit.
If the inference time is toolong, then the number of model calls will be limited, and as such, the number of explored routes willalso be limited.
This effect is evident in MHNreact and Chemformer, the models with the highestsearch times and lowest model calls across all experiments.
Single-step models typically evaluate in batches larger than one, however this does not currentlytransfer to multi-step retrosynthesis planning.
For example, MHNreact has the fastest inference timewhen using its default batch size, but its inference time is considerably increased when reducingthe batch size to one, the setting under which multi-step retrosynthesis planning is carried out.
Assuch, single-step models may not reach their full potential in search algorithms due to slow inferencetimes.
Noteworthy, most single-step models are developed for GPU use, and CPU usage can hindertheir inference speed.
However, it is necessary to conduct the multi-step experiments in parallelon CPUs due to the thousands of target molecules to be solved since massive GPU parallelizationis currently not available to the general research community.
Consequently, models designed andoptimized for the single-step prediction task may not perform well in multi-step retrosynthesisplanning.
Therefore, single-step model developers should take the potential multi-step applicationinto account and optimize their methods accordingly.
There are more general aspects to consider when discussing the divide between single-step andmulti-step retrosynthesis.
Though USPTO-50k is the most commonly used dataset for benchmarkingsingle-step retrosynthesis prediction models, it represents only a limited area of the chemical spacesuch that the models and our results may not apply to a more expansive chemical domain.
Moreover,
USPTO-50k comprises only single-step reactions, and the produced routes cannot be compared toreference routes to evaluate their validity.
Recently, new benchmarks have emerged to address thelack of multi-step reference data [29], so further work is required to quantify the produced routes.
Ideally, one would assess these routes irrespective of a particular reference route since there are manypotential valid routes for any target molecule.
However, at present, this can only be addressed by adomain expert, an extremely time-intensive task.
Additionally, this work focuses on using single-step models within two selected search algorithms.
However, other search approaches can also considerably impact multi-step retrosynthesis planningmethods [23, 24].
Thus finding the optimal combination of single-step and multi-step methods is yetto be explored and could have a substantial impact on synthesis planning in the future.
Conclusion
In this work, we bridge the gap between single-step retrosynthesis and multi-step retrosynthesisplanning.
By extending current state-of-the-art single-step models to the multi-step domain, we findno clear relationship between the single-step and multi-step benchmarking metrics, in particular singlestep top-k accuracy and multi-step solvability.
Additionally, we show the importance of developingsingle-step models for the multi-step domain, as single-step models can have an impressive impact onmulti-step retrosynthesis planning performance.
LocalRetro, the best performing single-step model,increases solvability by +30.0% to 80.6% and triples the number of solved routes compared to the most widely used model.
Interestingly, LocalRetro outperforms other single-step models, even thosewith similar single-step accuracy profiles.
Additionally, we analyze other potential factors involvedin the translation between the two domains, most notably the inference time of the single-step model.
Overall, we show there is no easy transfer of single-step retrosynthesis models to the multi-stepretrosynthesis planning domain.
With this work, we provide an overview of how current state-of-the-art single-step models fare withincontemporary search algorithms, however, we only evaluate a selected scope of single-step andmulti-step combinations.
In the future, more diverse chemical datasets need to be further explored toexamine the applicability of these approaches beyond the USPTO-50k dataset.
To summarize, we show that single-step models should be developed and tested for the multi-stepdomain, and not as an isolated task, to successfully identify synthesis routes for molecules of interest.