Original: Retrosynthesis Prediction Revisited
Revised: Retrosynthesis Prediction Revisited

Original: Anonymous Author s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: Retrosynthesis is an important problem in chemistry and represents an interesting challenge for AI since it involves predictions over sets of complex, moleculargraph structures.
Revised: Retrosynthesis is an important problem in chemistry and represents an interesting challenge for AI since it involves predictions over sets of complex, molecular graph structures.

Original: Recently, a wealth of models ranging from language models tograph neural networks are being proposed.
Revised: Recently, a wealth of models ranging from language models to graph neural networks are being proposed.

Original: However, most studies evaluate overa single dataset and split only, focus on top-1 accuracy, and provide few insightinto the actual capabilities of individual models.
Revised: However, most studies evaluate over a single dataset and split only, focus on top-1 accuracy, and provide few insight into the actual capabilities of individual models.

Original: This prevents research frommoving forward since issues to be addressed by future work are not identified.
Revised: This prevents research from moving forward since issues to be addressed by future work are not identified.

Original: In this paper, we focus on the evaluation: we show that the currently used datadoes not fit to test generalization, one of the main goals stated in the literature;propose new splits of the USPTO reactions modeling various scenarios; studyrepresentatives of the main types of models over this data; and finally present the,to the best of our knowledge, first evaluation and comparison of these modelsin the multi-step scenario.
Revised: In this paper, we focus on the evaluation: we show that the currently used data does not fit to test generalization, one of the main goals stated in the literature; propose new splits of the USPTO reactions modeling various scenarios; study representatives of the main types of models over this data; and finally present the, to the best of our knowledge, first evaluation and comparison of these models in the multi-step scenario.

Original: Altogether, we show that the picture is more diversethan the results over the usually used USPTO-50k data suggest.
Revised: Altogether, we show that the picture is more diverse than the results over the usually used USPTO-50k data suggest.

Original:  Introduction
Revised:  Introduction

Original: Retrosynthesis is an important problem in chemistry (Robinson, 1917; Corey & Wipke, 1969): tosynthesize a complex molecule in the lab based on purchasable molecules, a chemist needs toknow the sequence of reactions to run.
Revised: Retrosynthesis is an important problem in chemistry (Robinson, 1917; Corey & Wipke, 1969): to synthesize a complex molecule in the lab based on purchasable molecules, a chemist needs to know the sequence of reactions to run.

Original: The problem represents an interesting challenge for AIsince it involves predictions over molecular graph structures as well as a search component.
Revised: The problem represents an interesting challenge for AI since it involves predictions over molecular graph structures as well as a search component.

Original: Inevery step, a given product molecule (also, target ) is split into a set of reactant molecules , the targetsfor the next steps.
Revised: In every step, a given product molecule (also, target ) is split into a set of reactant molecules , the targets for the next steps.

Original: 1 Together, the steps form a retrosynthetic route , a tree structure.
Revised: 2 Together, the steps form a retrosynthetic route , a tree structure.

Original: The large number of models proposed for retrosynthesis prediction recently reflects the interest ofthe ML research community, for a good overview of the state of the art, we refer to the discussionsin recent works (Tu & Coley, 2021; Zhong et al., 2022).
Revised: The large number of models proposed for retrosynthesis prediction recently reflects the interest of the ML research community, for a good overview of the state of the art, we refer to the discussions in recent works (Tu & Coley, 2021; Zhong et al., 2022).

Original: The single-step models used for predictingthe individual steps can be categorized into three types: (I) Template-based approaches (Segler &
Revised: The single-step models used for predicting the individual steps can be categorized into three types: (I) Template-based approaches (Segler &

Original: Waller, 2017) classify a target w.r.t.
Revised: Waller, 2017) classify a target w.r.t.

Original: a set of reaction templates (i.e., rules how a target can be split ifit meets certain conditions), which are usually mined from the training data or manually encoded,and assumed to encode valid reactions.
Revised: a set of reaction templates (i.e., rules how a target can be split if it meets certain conditions), which are usually mined from the training data or manually encoded, and assumed to encode valid reactions.

Original: (II) Template-free approaches (Liu et al., 2017) operateon the data alone and usually consider the molecules in text-based SMILES representation andthe problem as a translation task (i.e., a sequence to sequence problem).
Revised: (II) Template-free approaches (Liu et al., 2017) operate on the data alone and usually consider the molecules in text-based SMILES representation and the problem as a translation task (i.e., a sequence to sequence problem).

Original: (III) Semi-templatebased approaches (Yan et al., 2020) apply a given atom mapping (i.e., considered as a kind oftemplate), relating the atoms in the targets to the corresponding ones in the reactants, during training.
Revised: (III) Semi-template based approaches (Yan et al., 2020) apply a given atom mapping (i.e., considered as a kind of template), relating the atoms in the targets to the corresponding ones in the reactants, during training.

Original: Current approaches in this category model a two-step procedure first identifying thereaction center (i.e., where to split), and then completing the obtained molecule parts to reactants.
Revised: Current approaches in this category model a two-step procedure first identifying the reaction center (i.e., where to split), and then completing the obtained molecule parts to reactants.

Original: There are a few open-source tools for multi-step synthesis planning (Genheden et al., 2020; ask;
Revised: There are a few open-source tools for multi-step synthesis planning (Genheden et al., 2020; ask;

Original: Chen et al., 2020).
Revised: Chen et al., 2020).

Original: They apply a template-based MLP for the single steps by default, and resort to
Revised: They apply a template-based MLP for the single steps by default, and resort to

Original: Retro ∗ (Chen et al., 2020) (i.e., a variant of A ∗ search) or MCTS for the multi-step planning.
Revised: Retro ∗ (Chen et al., 2020) (i.e., a variant of A ∗ search) or MCTS for the multi-step planning.

Original: Studies about single-step models primarily compare on the USPTO-50k dataset (Schneider et al.,2016), since many works report numbers only for that data, and a single (often random) split.
Revised: Studies about single-step models primarily compare on the USPTO-50k dataset (Schneider et al., 2016), since many works report numbers only for that data, and a single (often random) split.

Original: This dataset was extracted from the United States patent data from 1976 to 2016 (Lowe, 2017).
Revised: This dataset was extracted from the United States patent data from 1976 to 2016 (Lowe, 2017).

Original: While the latter represents one of the few large open sources for reaction data, we note that mostreactions (about 94%) in the USPTO-50k test set match templates one can extract from the trainingdata and hence do not truly fit to test generalization.
Revised: While the latter represents one of the few large open sources for reaction data, we note that most reactions (about 94%) in the USPTO-50k test set match templates one can extract from the training data and hence do not truly fit to test generalization.

Original: Yet many of the template-free proposalsconsider the ability to generalize as main advantage of their approaches (Liu et al., 2017; Somnathet al., 2021).
Revised: Yet many of the template-free proposals consider the ability to generalize as main advantage of their approaches (Liu et al., 2017; Somnath et al., 2021).

Original: Moreover, there is a strong focus on improving top-1 accuracy, which has beencritisized as being insufficient in the chemistry community (Schwaller et al., 2019).
Revised: Moreover, there is a strong focus on improving top-1 accuracy, which has been critisized as being insufficient in the chemistry community (Schwaller et al., 2019).

Original: Generally,few works provide insight into the actual capabilities of individual models and components, andconcrete comparisons between models (i.e., in terms of predictions) are rare, possibly becausemany implementations are less accessible.
Revised: Generally, few works provide insight into the actual capabilities of individual models and components, and concrete comparisons between models (i.e., in terms of predictions) are rare, possibly because many implementations are less accessible.

Original: The latter also often ignore practical aspects, suchthat applying the models out of the box would be hard.
Revised: The latter also often ignore practical aspects, such that applying the models out of the box would be hard.

Original: For example, most language-basedmodels operate in batch mode reading and writing from files (Zheng et al., 2019; Chen et al. and others require involved preprocessing or training procedures (Dai et al., 2019; Yan et al.,2020).
Revised: For example, most language-based models operate in batch mode reading and writing from files (Zheng et al., 2019; Chen et al., 2019), and others require involved preprocessing or training procedures (Dai et al., 2019; Yan et al., 2020).

Original: Altogether, this prevents research from moving forward since issues that are critical forapplicability are not identified, and research is rarely translated into applications.
Revised: Altogether, this prevents research from moving forward since issues that are critical for applicability are not identified, and research is rarely translated into applications.

Original: Related Work.
Revised: Related Work.

Original: While we above describe the general trend, there are certainly exceptions.
Revised: While we above describe the general trend, there are certainly exceptions.

Original: Inparticular, several analyses in the chemistry domain provide more detail, in terms of all aspects,data, methodology, and also in measuring effectiveness in the multi-step scenario.
Revised: In particular, several analyses in the chemistry domain provide more detail, in terms of all aspects, data, methodology, and also in measuring effectiveness in the multi-step scenario.

Original: Data.
Revised: Data.

Original: Somestudies also evaluate over larger USPTO subsets (Dai et al., 2019) or use different splits (Seidl et al. ,2022; Chen et al., 2019).
Revised: Some studies also evaluate over larger USPTO subsets (Dai et al., 2019) or use different splits (Seidl et al., 2022; Chen et al., 2019).

Original: Methodology.
Revised: Methodology.

Original: Recently, (Lin et al., 2022) proposed to re-rank predictionsand, in the course of this, compared different single-step models and, specifically, also in terms oftheir actual predictions.
Revised: Recently, (Lin et al., 2022) proposed to re-rank predictions and, in the course of this, compared different single-step models and, specifically, also in terms of their actual predictions.

Original: Further, (Chen et al., 2019; Lin et al., 2022) distinguish the predictionsin terms of reaction types.
Revised: Further, (Chen et al., 2019; Lin et al., 2022) distinguish the predictions in terms of reaction types.

Original: There have been further various proposals for metrics beyond top-k(Schwaller et al., 2019, 2020; Tetko et al., 2020; Lin et al., 2022)
Revised: There have been further various proposals for metrics beyond top-k.

Original: (see Appendix A).
Revised: (Schwaller et al., 2019, 2020) propose round-trip accuracy , the agreement between two models predicting the reaction in the forward and, respectively, backward direction; coverage , quantifying for how many of the products at least one valid suggestion of the set of reactants could be found; and diversity , the number of diverse, valid reactants after removing the buyable ones.

Original:  Multi-step.
Revised: Regarding diversity, they further consider a metric of statistical significance. (Tetko et al., 2020) propose to report MaxFrag , the recognition of the largest reactant, to estimate the ability of a model to deduce the correct reaction class; note the similarity to the coverage from (Schwaller et al., 2020). Most recently, (Lin et al., 2022) applied MRR , observing that it reflects a trend in between the ones of top1 and top-3 accuracy, and area under the top-k curve , which parallels top-k accuracy; while both are similar to the latter in trend, especially MRR offers better explainability in that it summarizes the latter for different k. Multi-step.

Original: Lastly,some works apply the proposed single-step model in the muti-step planning setting by eitherchaining single-step predictions (Coley et al., 2017) or combined with a search algorithm (Ishidaet al., 2022) but, to the best of our knowledge, there are no comparisons between models.
Revised: Lastly, some works apply the proposed single-step model in the muti-step planning setting by either chaining single-step predictions (Coley et al., 2017) or combined with a search algorithm (Ishida et al., 2022) but, to the best of our knowledge, there are no comparisons between models.

Original: Sucha comparison has come closer, with the recently proposed PARoutes framework (Genheden & Bjerrum, 2022), which offers tools for extracting synthetic routes from the USPTO patent data,based on the work of (Mo et al., 2021), and for comparing predicted routes to the extracted ones.(Genheden & Bjerrum, 2022) also propose two route datasets, n1 and n5, and present an evaluationusing template-based single-step models.
Revised: Such a comparison has come closer, with the recently proposed PARoutes framework (Genheden & Bjerrum, 2022), which offers tools for extracting synthetic routes from the USPTO patent data, based on the work of (Mo et al., 2021), and for comparing predicted routes to the extracted ones. (Genheden & Bjerrum, 2022) also propose two route datasets, n1 and n5, and present an evaluation using template-based single-step models.

Original: However, this route data was designed for evaluatingmodel capabilities under rather controlled conditions, it does not contain rare reactions andthe training data is in a certain sense artificial (exactly three reaction examples per reactiontemplate).
Revised: However, this route data was designed for evaluating model capabilities under rather controlled conditions, it does not contain rare reactions and the training data is in a certain sense artificial (exactly three reaction examples per reaction template).

Original: Therefore, we used this framework for the creation of more general datasets, and inour evaluation of the multi-step setting.
Revised: Therefore, we used this framework for the creation of more general datasets, and in our evaluation of the multi-step setting.

Original: There are some other route datasets: (Chen et al., 2020)also extracted routes from the USPTO data that do not contain rare reactions, and (Ishida et al.,2022) manually curated routes.
Revised: There are some other route datasets: (Chen et al., 2020) also extracted routes from the USPTO data that do not contain rare reactions, and (Ishida et al., 2022) manually curated routes.

Original: In summary, there are many reasonable proposals to evaluatesingle-step retrosynthesis models, yet these have not been picked up by the broader communityand are not commonly used today.
Revised: In summary, there are many reasonable proposals to evaluate single-step retrosynthesis models, yet these have not been picked up by the broader community and are not commonly used today.

Original: In the words of (Schwaller et al., 2019), the evaluation ofsingle-step retrosynthetic models is an overlooked research topic .
Revised: In the words of (Schwaller et al., 2019), the evaluation of single-step retrosynthetic models is an overlooked research topic .

Original: Our study focuses on the evaluation of single-step models, applies some of these ideas, demonstrates their relevance for applications, and complements the above works by focusing on important aspects that have been less in focus so far, namely, prediction diversity, generalization, andperformance in the multi-step scenario.
Revised: Our study focuses on the evaluation of single-step models, applies some of these ideas, demonstrates their relevance for applications, and complements the above works by focusing on important aspects that have been less in focus so far, namely, prediction diversity, generalization, and performance in the multi-step scenario.

Original: Our contributions are the following.
Revised: Our contributions are the following.

Original: • We propose new splits of the USPTO reactions.
Revised: • We propose new splits of the USPTO reactions.

Original: Since prediction diversity is important in viewof the incomplete reaction data (e.g., missing conditions which may restrict applicability) andthe models not being perfect, we collected products which can be synthesized from differentreactant sets and thus have multiple solutions in the test set USPTO-ms .
Revised: Since prediction diversity is important in view of the incomplete reaction data (e.g., missing conditions which may restrict applicability) and the models not being perfect, we collected products which can be synthesized from different reactant sets and thus have multiple solutions in the test set USPTO-ms .

Original: Moreover, we createdatasets USPTO-rt and USPTO-rd containing (the reactions of) retrosynthetic routes such thatwe can compare the performance in the single and multi-step scenarios by either evaluatingsingle-step models on the reactions with all reactions’ targets as input, or by running theminside a synthesis planning tool with only the roots of the routes as input.
Revised: Moreover, we create datasets USPTO-rt and USPTO-rd containing (the reactions of) retrosynthetic routes such that we can compare the performance in the single and multi-step scenarios by either evaluating single-step models on the reactions with all reactions’ targets as input, or by running them inside a synthesis planning tool with only the roots of the routes as input.

Original: This comparisonshows that there is indeed a discrepancy between the performance in the two scenarios.•
Revised: This comparison shows that there is indeed a discrepancy between the performance in the two scenarios.

Original:  In addition, we develop the multi-step score (mss) for estimating the multi-step performancebased on the predictions on individual reactions, independently of a route search algorithm.•
Revised: • In addition, we develop the multi-step score (mss) for estimating the multi-step performance based on the predictions on individual reactions, independently of a route search algorithm.

Original:  We evaluate several representatives of the main models types over various datasets and presentthe, to the best of our knowledge, first multi-step evaluation and comparison of these models.• To address issue of missing comparisons, we created all our test sets such that they do notoverlap with the USPTO-50k train and validation data (based on the usually used split from(Coley et al., 2017; Dai et al., 2019)).
Revised: • We evaluate several representatives of the main models types over various datasets and present the, to the best of our knowledge, first multi-step evaluation and comparison of these models. • To address issue of missing comparisons, we created all our test sets such that they do not overlap with the USPTO-50k train and validation data (based on the usually used split from (Coley et al., 2017; Dai et al., 2019)).

Original: Hence, researchers can use the existing checkpointsand evaluate easily over our new test sets.
Revised: Hence, researchers can use the existing checkpoints and evaluate easily over our new test sets.

Original: For the evaluation, we created a simple, abstractwrapper class for single-step retrosynthesis models; evaluation scripts based on this wrapper;and example wrappers for the models we evaluate in this work.
Revised: For the evaluation, we created a simple, abstract wrapper class for singlestep retrosynthesis models; evaluation scripts based on this wrapper; and example wrappers for the models we evaluate in this work.

Original: Furthermore, we provide anextension of AIZynthFinder (Genheden et al., 2020), such that it can be used directly with ourwrapper interface.
Revised: Furthermore, we provide an extension of AIZynthFinder (Genheden et al., 2020), such that it can be used directly with our wrapper interface.

Original: All assets and code will be made available.
Revised: All assets and code are provided at https://github.ibm.com/vthost/retroeval . The fast adoption of

Original: The fast adoption of Retro ∗ (Genheden et al., 2020; Kim et al., 2021) has shown that the communityis open to adopt latest research outcomes if they are accessible, efficient, and effective.
Revised: Retro ∗ (Genheden et al., 2020; Kim et al., 2021) has shown that the community

Original: The goal of our work is  to ease the development of single-step models towards this direction.
Revised: is open to adopt latest research outcomes if they are accessible, efficient, and effective

Revised: The

Revised: goal of

Revised: our work is to

Revised: ease the development of

Revised: single-step models towards this direction.

Original:  Our Setting
Revised:  Our Setting

Original: Models considered in this paper.
Revised: Models considered in this paper.

Original: • Template-based:
Revised: • Template-based:

Original: NeuralSym (NPP) (Segler & Waller, 2017), a regular MLP; the ConditionalGraph Logic Network (GLN) (Dai et al., 2019), which models template applicability using a conditional graphical model that is parameterized using a structure-based embedding; and the recently proposed MHNreact (MHN) (Seidl et al., 2022), which uses a modern Hopfield network (Widrich et al., 2020) to retrieve stored templates based on the input molecule embedding.
Revised: NeuralSym (NPP) (Segler & Waller, 2017), a regular MLP; the Conditional Graph Logic Network (GLN) (Dai et al., 2019), which models template applicability using a conditional graphical model that is parameterized using a structure-based embedding; and the recently proposed MHNreact (MHN) (Seidl et al., 2022), which uses a modern Hopfield network (Widrich et al., 2020) to retrieve stored templates based on the input molecule embedding.

Original: Both GLN and MHN also encode the templates and hence can exploit similarities between them.
Revised: Both GLN and MHN also encode the templates and hence can exploit similarities between them.

Original: • Template-free: Chemformer (CF) and Chemformer-Large (CFL) (Irwin et al., 2022), regular BART language models (Lewis et al., 2020) that were pre-trained on a large molecule corpus and only differ in size; and Graph2SMILES (G2S)
Revised: • Template-free: Chemformer (CF) and Chemformer-Large (CFL) (Irwin et al., 2022), regular BART language models (Lewis et al., 2020) that were pre-trained on a large molecule corpus and only differ in size; and Graph2SMILES (G2S)

Original: (Tu & Coley, 2021), combining a graph-based transformer encoder with a sequence-based decoder.
Revised: (Tu & Coley, 2021), combining a graph-based transformer encoder with a sequence-based decoder.

Original: • Semi-template-based:
Revised: • Semi-template-based:

Original: GraphRetro (GR) (Somnath et al., 2021), modeling intermediate reactionsteps with GNNs.
Revised: GraphRetro (GR) (Somnath et al., 2021), modeling intermediate reaction steps with GNNs.

Original: Note that we tried to set up other models, especially from the last category, but often failed.
Revised: Note that we tried to set up other models, especially from the last category, but often failed.

Original: Further, we were able to run GLN only locally and therefore present less results for this model.
Revised: Further, we were able to run GLN only locally and therefore present less results for this model.

Original: Metrics.
Revised: Metrics.

Original: Top-k accuracy ( top-k ) represents the fraction of test targets for which the correct reactantset is among the top-k ranked predicted reactant sets.
Revised: Top-k accuracy ( top-k ) represents the fraction of test targets for which the correct reactant set is among the top-k ranked predicted reactant sets.

Original: Note that the former is usually used in asetting where a target with multiple correct reactant sets is considered as two different test pointsand the model is evaluated twice w.r.t.
Revised: Note that the former is usually used in a setting where a target with multiple correct reactant sets is considered as two different test points and the model is evaluated twice w.r.t.

Original: the two solutions, respectively.
Revised: the two solutions, respectively.

Original: We further consider themean reciprocal rank ( MRR ) and MaxFrag accuracy at k ( mf-k ), focusing on the recognition ofthe largest reactant.
Revised: We further consider the mean reciprocal rank ( MRR ) and MaxFrag accuracy at k ( mf-k ), focusing on the recognition of the largest reactant.

Original: In the scenario where we focus on predicting multiple solutions, we consider 
Revised: In the scenario where we focus on predicting multiple solutions, we consider recall@k ( r@k ), which represents the proportion of correct reactant sets among the top-k ranked predictions averaged [Table]

Original: [Table]
Revised: [Table]

Original: # T a r g e t s
Revised: # T a r g e t s

Original: Reactions
Revised: Reactions

Original: [Table]
Revised: [Table]

Revised: target

Original: TemplateOther
Revised: TemplateOther

Original: [Table]
Revised: [Table]

Original:  Revisiting USPTO-50k
Revised:  Revisiting USPTO-50k

Original: In most experiments, we consider the models trained over USPTO-50k (or short, 50k).
Revised: In most experiments, we consider the models trained over USPTO-50k (or short, 50k).

Original: If available,we used the checkpoints from the original studies within our wrappers.
Revised: If available, we used the checkpoints from the original studies within our wrappers.

Original: Some of the trends weobserve turned out to be more general (see also Sec. 6 and Appendix C):
Revised: Some of the trends we observe turned out to be more general (see also Sec. 6 and Appendix B):

Original: • top-1: CF(L) is best, followed by (in no particular order) G2S and GR.
Revised: • top-1: CF(L) is best, followed by (in no particular order) G2S and GR.

Original: • MRR, top-5-50: Template-based models are best, followed by (in no particular order)
Revised: • MRR, top-5-50: Template-based models are best, followed by (in no particular order)

Original: G2S, GR.
Revised: G2S, GR.

Original: • G2S performs more similar to GR than to CF(L).
Revised: • G2S performs more similar to GR than to CF(L).

Original: CF-L usually outperforms CL slightly.
Revised: CF-L usually outperforms CL slightly.

Original: • top-50: GLN/MHN basically reach maximal possible performance given the amountof test
Revised: • top-50: GLN/MHN basically reach maximal possible performance given the amount

Revised: of test

Original: reactions matching templates (see Figure 4).
Revised: reactions matching templates (see Figure 4)

Original: In particular, they are already close to that attop-
Revised: In particular they are already close to that at

Revised: top-

Original: Note that, the template-based top-k predictions are selected from those templates that areactually applicable (i.e., their conditions are satisfied by the target) but our NeuralSym and MHN implementations pre-compute only the applicability of the first 200 templates; whenconsidering all, MHN reaches 93.2% at top-50.• MRR is
Revised: Note that, the template- based top-k predictions are selected from those templates that are actually applicable (i.e., their conditions are satisfied by the target) but our NeuralSym and MHN implementations pre-compute only the applicability of the first 200 templates; when considering all, MHN reaches 93.2% at top-50.

Original:  between top-1 and top-5, and mf- is more similar to top- than to top-
Revised: • MRR is between top-1 and top-5, and mf-10 is more similar to top-50 than to top-10.

Original:  Prediction Diversity: The Dataset USPTO-ms
Revised:  Prediction Diversity: The Dataset USPTO-ms

Original: USPTO-ms.
Revised: USPTO-ms.

Original: For evaluating how well the models capture diversity, we collected reactions fromthe USPTO patent data (Lowe, 2017) where we have multiple solutions (i.e., multiple possiblereactant sets) into the new test set USPTO-ms. Details about the creation process can be foundin Appendix D.
Revised: For evaluating how well the models capture diversity, we collected reactions from the USPTO patent data (Lowe, 2017) where we have multiple solutions (i.e., multiple possible reactant sets) into the new test set USPTO-ms. Details about the creation process can be found in Appendix C.

Original: The data contains 3,501 different targets and 7,438 different reactions.
Revised: The data contains 3,501 different targets and 7,438 different reactions.

Original: Figure 1shows further statistics.
Revised: Figureshows further statistics.

Original: Note that we dropped reactions with more than six solutions since thesewere extremely rare.
Revised: Note that we dropped reactions with more than six solutions since these were extremely rare.

Original: We see that only about a quarter of the reactions match templates from the
Revised: We see that only about a quarter of the reactions match templates from the

Original: USPTO-50k train/valid data, hence it is not only the diversity but also the novelty of the reactionswhich represents a challenge for the models.
Revised: USPTO-50k train/valid data, hence it is not only the diversity but also the novelty of the reactions which represents a challenge for the models.

Original: It is possible that the latter is the greater challenge.
Revised: It is possible that the latter is the greater challenge.

Original: #Solutionsintop # T a r g e t s ( % )
Revised: [Figure]

Original: NPP
Revised: NPP

Original: GLN
Revised: GLN

Original: MHN
Revised: MHN

Original: CFL
Revised: CFL

Original: G2S
Revised: G2S

Original: GR
Revised: GR

Original: Results, Table 1, Fig. 2.
Revised: Results, Table 1, Fig. 2.

Original: At first glance, the picture is very differentfrom the one over USPTO-50k.
Revised: At first glance, the picture is very different from the one over USPTO-50k.

Original: In particular, we observe no considerable differences between the models at r@1, with the exception of
Revised: In particular, we observe no considerable differences between the models at r@1, with the exception of

Original: NPP.
Revised: NPP.

Original: Note that we include the scores on the test data to show the maximal r@k possible.
Revised: Note that we include the scores on the test data to show the maximal r@k possible.

Original: Further, good scores at k = 1..3 are especially hardto obtain because most targets have two or three solutions.
Revised: Further, good scores at k = 1..3 are especially hard to obtain because most targets have two or three solutions.

Original: At higherk’s it gets again easier for the models since they allow for mistakes.
Revised: At higher k’s it gets again easier for the models since they allow for mistakes.

Original: Here, the template-based models and G2S perform especially well.
Revised: Here, the template-based models and G2S perform especially well.

Original: Since the table shows the average across targets, Figure 2, additionallyshows how many solutions the models actually find per target among 
Revised: Since the table shows the average across targets, Figure 2, additionally shows how many solutions the models actually find per target among the top 5.

Revised: We see that the r@k performance can be largely attributed to single solutions.

Revised: In particular, CF(L) seems to lack diversity.

Original: #Reactions
Revised: #Reactions

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        100.
Revised: oo d        100.

Original: #Longestlinearroute train validtest rt-1k        100.
Revised: #Longestlinearroute train validtest rt-1k        100.

Original: #Reactions
Revised: #Reactions

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        100.
Revised: oo d        100.

Original: #Longestlinearroute train validtest rd-1k
Revised: #Longestlinearroute train validtest rd-1k

Original: [Figure]
Revised: [Figure]

Original: the top 5.

Original: We see that the r@k performance can be largely attributedto single solutions.

Original: In particular, CF(L) seems to lack diversity.

Original:  On Evaluating Single & Multi-Step Prediction
Revised:  On Evaluating Single & Multi-Step Prediction

Original: Prediction in Route Context, the Datasets rt & rd, Figure 3, Table 7.
Revised: Prediction in Route Context, the Datasets rt & rd, Figure 3, Table 7.

Original: We followed (Mo et al., 2021;
Revised: We followed (Mo et al., 2021;

Original: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.
Revised: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.

Original: However,we did not drop rare reactions and created two custom splits.
Revised: However, we did not drop rare reactions and created two custom splits.

Original: In particular, we created a time split,the dataset USPTO-rt (or short, rt ), and one where we extracted sets of very diverse routes as testand validation data and kept all remaining routes as train data, USPTO-rd (or short, rd ).
Revised: In particular, we created a time split, the dataset USPTO-rt (or short, rt ), and one where we extracted sets of very diverse routes as test and validation data and kept all remaining routes as train data, USPTO-rd (or short, rd ).

Original: Furtherdetails about the creation are given in Appendix E.
Revised: Further details about the creation are given in Appendix D.

Original: The figures show that the rt data containstrain/valid/test distributions that are very similar, while there is great diversity in rd.
Revised: The figures show that the rt data contains train/valid/test distributions that are very similar, while there is great diversity in rd.

Original: The latterdataset contains more data overall because the test and validation routes are longer and we keptthe data from all other patents for training.
Revised: The latter dataset contains more data overall because the test and validation routes are longer and we kept the data from all other patents for training.

Original: For rt we dropped all remaining data from patentsat the test time points.
Revised: For rt we dropped all remaining data from patents at the test time points.

Original: We will use these full rt and rd datasets particularly to compare how themodel performance changes when trained over data beyond USPTO-50k.
Revised: We will use these full rt and rd datasets particularly to compare how the model performance changes when trained over data beyond USPTO-50k.

Original: Test Subsets: rt-1k & rd-1k, Figure 7, Table 7.
Revised: Test Subsets: rt-1k & rd-1k, Figure 7, Table 7.

Original: Since the models turned out to be more resourceand especially time-intensive than the regularly used MLPs in the multi-step scenario, we extractedsubsets of the original rt and rd test sets which will be in the focus of our study.
Revised: Since the models turned out to be more resource and especially time-intensive than the regularly used MLPs in the multi-step scenario, we extracted subsets of the original rt and rd test sets which will be in the focus of our study.

Original: Each contains(the reactions of) 1k routes.
Revised: Each contains (the reactions of) 1k routes.

Original: We additionally ensured that all reactions are unique, that we havemaximally one route per patent, and that there is no overlap with the USPTO-50k train/valid data.
Revised: We additionally ensured that all reactions are unique, that we have maximally one route per patent, and that there is no overlap with the USPTO-50k train/valid data.

Original: The overview of our test sets compared to the ones from (Genheden & Bjerrum, 2022) in the figureshows that the statistics are similar.
Revised: The overview of our test sets compared to the ones from (Genheden & Bjerrum, 2022) in the figure shows that the statistics are similar.

Original: For comparison, we also evaluated over subsets of n1 and n5,but did not observe considerable differences; hence the latter lie primarily in the nature of thetraining data (i.e., the n1/n5 train data is more similar to the test data).
Revised: For comparison, we also evaluated over subsets of n1 and n5, but did not observe considerable differences; hence the latter lie primarily in the nature of the training data (i.e., the n1/n5 train data is more similar to the test data).

Original: The Multi-Step Score (mss).
Revised: The Multi-Step Score (mss).

Original: In a nutshell, mss counts targets only then as solved if the targetsoccurring previously in the route are solved as well; see Appendix G for a formal definition.
Revised: In a nutshell, mss counts targets only then as solved if the targets occurring previously in the route are solved as well; see Appendix F for a formal definition.

Original: Wewill see that mss nicely complements existing metrics.
Revised: We will see that mss nicely complements existing metrics.

Original: [Table]

Original:  Evaluation in Route Context
Revised:  Evaluation in Route Context

Original: Single-Step Results on 1k Test Sets, Tables 2, Figure 5.
Revised: Single-Step Results on 1k Test Sets, Tables 2, Figure 5.

Original: Our evaluation over all test setsgenerally confirms the trends outlined in Section 3 in the context of USPTO50k, including the additional results on n1 and n5 in Appendix H. Still, there are notable differences.
Revised: Our evaluation over all test sets generally confirms the trends outlined in Section 3 in the context of USPTO50k, including [Table]

Original: [Figure]
Revised: [Figure]

Revised: the additional results on n1 and n5 in Appendix G.

Revised: Still, there are notable differences.

Original: The numbers are much lower, showing that the data from (Genheden& Bjerrum, 2022) as well as ours is more challenging.
Revised: The numbers are much lower, showing that the data from (Genheden & Bjerrum, 2022) as well as ours is more challenging.

Original: More interestingly, at top-1, the template-based model MHN is basically en parwith G2S and GR.
Revised: More interestingly, at top-1, the template-based model MHN is basically en par with G2S and GR.

Original: The Chemformers are clearly winning but considerably worse at higher k’s.
Revised: The Chemformers are clearly winning but considerably worse at higher k’s.

Original: The results on all datasets are in similarranges with the ones for rd-1k being lowest.
Revised: The results on all datasets are in similar ranges with the ones for rd-1k being lowest.

Original: In Figure 5, we cansee that the performance can be attributed primarily to the targetsmatching templates seen during training.
Revised: In Figure 5, we can see that the performance can be attributed primarily to the targets matching templates seen during training.

Original: There are only few signals,although there are some, that the semi/non-template-based modelsare capable to generalize, especially G2S and GR.
Revised: There are only few signals, although there are some, that the semi/non-templatebased models are capable to generalize, especially G2S and GR.

Original: Our mss score differs considerably in magnitude.
Revised: Our mss score differs considerably in magnitude.

Original: Interestingly, its trend is very similarto top-k for k > 1, although it is interpreted differently.
Revised: Interestingly, its trend is very similar to top-k for k > 1, although it is interpreted differently.

Original: Neverthelessnote that it better reflects the route context, in that the numbers arelowest on the more complex routes (rd and n5) while, for example,top-5 on n5-1k is higher than on n1-1k; and the difference to mss onthe routes in rt is larger than, for instance, the differences at MRR.
Revised: Nevertheless note that it better reflects the route context, in that the numbers are lowest on the more complex routes (rd and n5) while, for example, top-5 on n5-1k is higher than on n1-1k; and the difference to mss on the routes in rt is larger than, for instance, the differences at MRR.

Original: Comparison of Predictions, Figure 6.
Revised: Comparison of Predictions, Figure 6.

Original: We conclude the analysis of the single-step models trainedon USPTO-50k by comparing the actual predictions, w.r.t.
Revised: We conclude the analysis of the single-step models trained on USPTO-50k by comparing the actual predictions, w.r.t.

Original: a top-5 threshold, in terms of thepairwise agreement between models on solutions, averaged over all six datasets, and the exactopposite, the percentage of solutions predicted by a single model only, w.r.t.
Revised: a top-5 threshold, in terms of the pairwise agreement between models on solutions, averaged over all six datasets, and the exact opposite, the percentage of solutions predicted by a single model only, w.r.t.

Original: the total number ofthose solutions per dataset.
Revised: the total number of those solutions per dataset.

Original: The former shows that there seems to be a correlation within a class of models.
Revised: The former shows that there seems to be a correlation within a class of models.

Original: While the absolute results show greater similarity between G2S and GR (i.e., instead of
Revised: While the absolute results show greater similarity between G2S and GR (i.e., instead of

Original: G2S and CF(L)), here it is slightly larger between G2S and CFL, also between the template-basedmodels.
Revised: G2S and CF(L)), here it is slightly larger between G2S and CFL, also between the template-based models.

Original: Note that (Lin et al., 2022) provide a similar but more fine-grained analysis comparing theactual ranks of predictions from different models and show that these may differ. The right pictureshows that the usually best models, CFL and GLN w.r.t. top-1 and MRR/top-5, respectively, areinterestingly indeed often the ones which predict solutions that are not predicted by other models.
Revised: Note that (Lin et al., 2022) provide a similar but more fine

Original: Scaling Up the Train Data, Tables 11, 12.
Revised: Scaling Up the Train Data, Tables 11, 12.

Original: On the 1k test sets we can compare to the modelstrained over USPTO-50k (see Table 2).
Revised: On the 1k test sets we can compare to the models trained over USPTO-50k (see Table 2).

Original: The impact of retraining turns out to be much higherfor rd-1k than for rt-1k; for example, we have a top-1 increase of about 10%, vs. 3-5% on rt-1k.
Revised: The impact of retraining turns out to be much higher for rd-1k than for rt-1k; for example, we have a top-1 increase of about 10%, vs. 3-5% on rt-1k.

Original: This might be explained by the larger amount of training data in rd-1k and, in particular, by thegreater similarity between train and test data.
Revised: This might be explained by the larger amount of training data in rd-1k and, in particular, by the greater similarity between train and test data.

Original: Recall that we have strict temporal split with rt.
Revised: Recall that we have strict temporal split with rt.

Original: One very likely explanation, and consequence of the similarity, is the increase in test reactionsmatching templates seen during training.
Revised: One very likely explanation, and consequence of the similarity, is the increase in test reactions matching templates seen during training.

Original: Figure 4, shows that the share for rd is now larger thanthe one for rt.
Revised: Figure 4, shows that the share for rd is now larger than the one for rt.

Original: Apparently, this impacts the performance of the template-free CF nearly as muchas the template-based models.
Revised: Apparently, this impacts the performance of the template-free CF nearly as much as the template-based models.

Original: Finally, we note that there is no gap anymore between the latter;in fact, for larger k’s, which we consider in the multi-step setting, NPP seems to provide betterperformance.
Revised: Finally, we note that there is no gap anymore between the latter; in fact, for larger k’s, which we consider in the multi-step setting, NPP seems to provide better performance.

Original: The results on the full test sets are very similar and shown in Table 12.
Revised: The results on the full test sets are very similar and shown in Table 12.

Original: The Multi-Step Scenario, Table 3.
Revised: The Multi-Step Scenario, Table 3.

Original: Our results give insights into the models’ out-of-the-boxusability.
Revised: Our results give insights into the models’ out-of-the-box usability.

Original: We show the top-5 accuracy w.r.t.
Revised: We show the top-5 accuracy w.r.t.

Original: finding the exact test route.
Revised: finding the exact test route.

Original: Since this criterion israther strong, we also report the maximal leaves overlap with this route within the top-k predictedroutes and consider a subset of 100 routes where all steps match templates seen during training.
Revised: Since this criterion is rather strong, we also report the maximal leaves overlap with this route within the top-k predicted routes and consider a subset of 100 routes where all steps match templates seen during training.

Original: Note that, interestingly, there is basically no difference between the top-5 and top-10 scores,so that we only show the former here.
Revised: Note that, interestingly, there is basically no difference between the top-5 and top-10 scores, so that we only show the former here.

Original: At first glance, we see that the performances are verydifferent from the corresponding single-step experiments, and our selection of datasets revealsfurther differences.
Revised: At first glance, we see that the performances are very different from the corresponding single-step experiments, and our selection of datasets reveals further differences.

Original: Over the manually picked routes where all reactions match templates seenduring training, the template-based models show much better performance than all other models.
Revised: Over the manually picked routes where all reactions match templates seen during training, the template-based models show much better performance than all other models.

Original: The results for the other datasets are extremely low reflecting the struggle of the models withdistributions beyond the train data and more complex routes (esp.
Revised: The results for the other datasets are extremely low reflecting the struggle of the models with distributions beyond the train data and more complex routes (esp.

Original: rd).
Revised: rd).

Original: One main, additionalproblem we identified are the probabilities returned by the models, the search algorithms use thosefor cost estimates.
Revised: One main, additional problem we identified are the probabilities returned by the models, the search algorithms use those for cost estimates.

Original: In particular the template-free models based on common NLP frameworksreturn no normalized or regularized probabilities, assigning high ones to only to a few predictions,and hence prevent the search from exploring alternatives to their top suggestions altogether.
Revised: In particular the template-free models based on common NLP frameworks return no normalized or regularized probabilities, assigning high ones to only to a few predictions, and hence prevent the search from exploring alternatives to their top suggestions altogether.

Original: This is problematic given that their top-1 performance is not perfect.
Revised: This is problematic given that their top-1 performance is not perfect.

Original: We experimented withnormalization and obtained best results (the ones shown) for all models by applying an additionalsoftmax on the top-10 predictions given to AIZynthFinder.
Revised: We experimented with normalization and obtained best results (the ones shown) for all models by applying an additional softmax on the top-10 predictions given to AIZynthFinder.

Original: Note that (Tu & Coley, 2021) mentionthat additional engineering efforts are likely needed for G2S in the multi-step scenario, but theimpact of such adaptations on both single and multi-step performance remains unclear.
Revised: Note that (Tu & Coley, 2021) mention that additional engineering efforts are likely needed for G2S in the multi-step scenario, but the impact of such adaptations on both single and multi-step performance remains unclear.

Original: Theresults for G2S further reveal a shortcoming of the top-k evaluation in PARoutes: the routes aresorted including route length as one criterion but, in this way, very unlikely predictions may betaken as top ones, especially with the template-free models.
Revised: The results for G2S further reveal a shortcoming of the top-k evaluation in PARoutes: the routes are sorted including route length as one criterion but, in this way, very unlikely predictions may be taken as top ones, especially with the template-free models.

Original: We experimented with atom mappingor a forward-direction model as suggested in (Schwaller et al., 2020) to validate the outcomes ofthe template-free models, and hence filter out unlikely predictions, but this causes considerableoverhead.
Revised: We experimented with atom mapping or a forward-direction model as suggested in (Schwaller et al., 2020) to validate the outcomes of the template-free models, and hence filter out unlikely predictions, but this causes considerable overhead.

Original: In terms of metrics, we observe that the trends shown here are captured by most metricswith the exception of top-1 accuracy, and at different magnitudes.
Revised: In terms of metrics, we observe that the trends shown here are captured by most metrics with the exception of top-1 accuracy, and at different magnitudes.

Original: Our mss score is similar, butcaptures a different intent and some subtle difference (see previous subsection), in particularthe large gap between the routes in rt and rd.
Revised: Our mss score is similar, but captures a different intent and some subtle difference (see previous subsection), in particular the large gap between the routes in rt and rd.

Original: Overall, NPP performs surprisingly well and isonly outperformed by GLN; we could not find a specific reason for why it outperforms MHNbut hypothesize that it is due to better regularization capability.
Revised: Overall, NPP performs surprisingly well and is only outperformed by GLN; we could not find a specific reason for why it outperforms MHN but hypothesize that it is due to better regularization capability.

Original: Lastly, in comparison to theexperiments with the template-based models in (Genheden & Bjerrum, 2022), our results aremuch lower - as expected -, but also show much greater variation between the datasets.
Revised: Lastly, in comparison to the experiments with the template-based models in (Genheden & Bjerrum, 2022), our results are much lower - as expected -, but also show much greater variation between the datasets.

Original: This,again, seems to hint at a considerable impact of rare reactions, which occur more often in rd andturn out to be critical in its longer and more complex routes.[Table]
Revised: This, again, seems to hint at a considerable impact of rare reactions, which occur more often in rd and turn out to be critical in its longer and more complex routes.

Original: Discussion.
Revised: Discussion.

Original: • Generally, the identification of data aspects influencing predictions is critical for correctly inter- preting the results .
Revised: • Generally, the identification of data aspects influencing predictions is critical for correctly interpreting the results .

Original: We show that the templates from the train data reveal interesting insights
Revised: We show that the templates from the train data reveal interesting insights

Original: • The variety of models does not seem to lead to a large variety of predictions; in our experiments,the models predicted few truly unique solutions (i [Table] e., usually less than 1% of the test set).
Revised: • The variety of models does not seem to lead to a large variety of predictions; in our experiments, the models predicted few truly unique solutions (i.e., usually less than 1% of the test set).

Original: • The main available search algorithms for the multi-step setting expect scaled and regularizedprobabilities , which are not given by all models.
Revised: • The main available search algorithms for the multi-step setting expect scaled and regularized probabilities , which are not given by all models.

Original: Hence it is not clear how those can be appliedand how they perform.
Revised: Hence it is not clear how those can be applied and how they perform.

Original: Similarly, we need efficient techniques to estimate the validity of thepredictions of template-free models.•
Revised: Similarly, we need efficient techniques to estimate the validity of the predictions of template-free models. [Table]

Original:  Currently, the template-based models are competitive with other models and, in the multi-stepsetting, the most simple and easy to use MLP still seems to provide one of the best solutionssince it is both efficient and effective.
Revised: • Currently, the template-based models are competitive with other models and, in the multi-step setting, the most simple and easy to use MLP still seems to provide one of the best solutions since it is both efficient and effective.

Original: In our study, only GLN performs better.• There is certainly major value in studying and comparing methods in a more theoretical contextin the single-step scenario only, as it is usually done today.
Revised: In our study, only GLN performs better. • There is certainly major value in studying and comparing methods in a more theoretical context in the single-step scenario only, as it is usually done today.

Original: Our work supports such studiesby showing the variation and commonalities in metrics beyond top-k accuracy.
Revised: Our work supports such studies by showing the variation and commonalities in metrics beyond top-k accuracy.

Original: Moreover, wepropose the mss score, as a first attempt to capture other aspects relevant in route context.•
Revised: Moreover, we propose the mss score, as a first attempt to capture other aspects relevant in route context.

Original:  The main open problem in terms of template-based models is the improvement in ranking .While
Revised: • The main open problem in terms of template-based models is the improvement in ranking .

Original:  the latter is in focus of current research, this improvement should not happen at theexpense of top-k for higher k’s given that multi-step prediction is the ultimate goal.
Revised: While the latter is in focus of current research, this improvement should not happen at the expense of top-k for higher k’s given that multi-step prediction is the ultimate goal.

Original: Here, MRRoffers a good overall estimate, and the template-based models represent strong baselines.• Generalization beyond reactions similar to the ones seen during training represents the othermajor challenge and seems to be solved only partly to date.
Revised: Here, MRR offers a good overall estimate, and the template-based models represent strong baselines. • Generalization beyond reactions similar to the ones seen during training represents the other major challenge and seems to be solved only partly to date.

Original: This leads to particularly lowperformance in the multi-step scenario, where a single new reaction leads to an overall failure.
Revised: This leads to particularly low performance in the multi-step scenario, where a single new reaction leads to an overall failure.

Original: Our work is intended to point out critical open issues rather than as a study of the SOTA
Revised: Our work is intended to point out critical open issues rather than as a study of the SOTA.

Original: Consider- ing recently proposed add ons (Fortunato et al., 2020; Sun et al., 2021; Lin et al., 2022), the actual
Revised: Considering recently proposed add ons (Fortunato et al., 2020; Sun et al., 2021; Lin et al., 2022), the actual

Original: SOTA is likely slightly better.
Revised: SOTA is likely slightly better.

Original: We cannot fully generalize from our results because we focused onrepresentative models and data only, and the research prototypes we probed have not been optimized for the multi-step setting.
Revised: We cannot fully generalize from our results because we focused on representative models and data only, and the research prototypes we probed have not been optimized for the multi-step setting.

Original: Note that template-free models have been successfully integratedin tools with careful engineering (Schwaller et al., 2020), and advanced search algorithms can overcome shortcomings of single-step models; e.g., (Chen et al., 2020; Kim et al., 2021) obtain 
Revised: Note that template-free models have been successfully integrated in tools with careful engineering (Schwaller et al., 2020), and advanced search algorithms can overcome shortcomings of single-step models; e.g., (Chen et al., 2020; Kim et al., 2021) obtain close to maximal performance with NeuralSym combined with an optimized cost estimate.

Original: close to maximal performance with NeuralSym combined with an optimized cost

Original: estimate

Original:  Conclusions
Revised:  Conclusions

Original: We provide new datasets for retrosynthesis prediction which allow for evaluating predictiondiversity, effectiveness in the multi-step context, and generalization, aspects that are usually notaddressed when evaluating over USPTO-50k.
Revised: We provide new datasets for retrosynthesis prediction which allow for evaluating prediction diversity, effectiveness in the multi-step context, and generalization, aspects that are usually not addressed when evaluating over USPTO-50k.

Original: We further provide evaluation infrastructure.
Revised: We further provide evaluation infrastructure.

Original: Our evaluation of various single-step models suggests that the aspects our data addresses are criticalfor estimating multi-step performance.
Revised: Our evaluation of various single-step models suggests that the aspects our data addresses are critical for estimating multi-step performance.

Original: Altogether, our work shows that we need to evaluate more broadly to be able to address the challenges of retrosynthesis prediction in future research.
Revised: Altogether, our work shows that we need to evaluate more broadly to be able to address the challenges of retrosynthesis prediction in future research.

Original: A Relate Work on Metrics
Revised: Acknowledgments

Original: (Schwaller et al., 2019, 2020) propose round-trip accuracy , the agreement between two modelspredicting the reaction in the forward and, respectively, backward direction; coverage , quantifyingfor how many of the products at least one valid suggestion of the set of reactants could be found;and diversity , the number of diverse, valid reactants after removing the buyable ones. Regardingdiversity, they further consider a metric of statistical significance. (Tetko et al., 2020) proposeto report MaxFrag , the recognition of the largest reactant, to estimate the ability of a model todeduce the correct reaction class; note the similarity to the coverage from (Schwaller et al., 2020).

Original: Most recently, (Lin et al., 2022) applied MRR , observing that it reflects a trend in between theones of top-1 and top-3 accuracy, andarea under the top-k curve , which parallels top-k accuracy;while both are similar to the latter in trend, especially MRR offers better explainability in that itsummarizes the latter for different k.
Revised: This work started within a course project at UMass Amherst. We thank Sahasra Iyer for initial discussions, and the tutors, especially Jay-Yoon Lee, and Andrew McCallum for all their support.

Original: B Model Configuration
Revised: A Model Configuration

Original: Single Step Models.
Revised: Single Step Models.

Original: For all models, we used the configurations suggested by the authors in theoriginal works (see Section 2).
Revised: For all models, we used the configurations suggested by the authors in the original works (see Section 2).

Original: The details can also be found in our repository.
Revised: The details can also be found in our repository.

Original: NeuralSym(++).
Revised: NeuralSym(++).

Original: We ran hyperparameter tuning since the original paper (Segler & Waller, 2017)used different data; the final configuration(s) are detailed in the repository.
Revised: We ran hyperparameter tuning since the original paper (Segler & Waller, 2017) used different data; the final configuration(s) are detailed in the repository.

Original: We made some interesting observations in these experiments: (1) We also experimented with the more complex Highwaynetwork version described in (Segler & Waller, 2017), but the MLP provided better performance.(2)
Revised: We made some interesting observations in these experiments: (1) We also experimented with the more complex Highway network version described in (Segler & Waller, 2017), but the MLP provided better performance.

Original:  We needed to add batch normalization to reach a performance comparable to the one reportedin (Dai et al., 2019); this design choice was later confirmed in communication with chemists whohave experience with template-based models and noted the importance of regularization, inparticular, for the muti-step experiments.
Revised: (2) We needed to add batch normalization to reach a performance comparable to the one reported in (Dai et al., 2019); this design choice was later confirmed in communication with chemists who have experience with template-based models and noted the importance of regularization, in particular, for the muti-step experiments.

Original: (3) We also experimented with optimization, using thevalidation loss, top-1, top-5, and top-50 as criteria, respectively; interestingly, all these scenarios lead to basically the same hyperparameters (i.e., there was only some slight variation in the fingerprint dimension, 1028 vs. 2048, and layer number, 2 vs. 3).
Revised: (3) We also experimented with optimization, using the validation loss, top-1, top-5, and top-50 as criteria, respectively; interestingly, all these scenarios lead to basically the same hyperparameters (i.e., there was only some slight variation in the fingerprint dimension, 1028 vs. 2048, and layer number, 2 vs. 3).

Original: It is left open if this is due to thesimplicity of the model or also holds for more complex ones.
Revised: It is left open if this is due to the simplicity of the model or also holds for more complex ones.

Original: Lastly, note that for the experimentsover the larger rt and rd data, we disregarded templates for which there is only one instance in thetraining data.
Revised: Lastly, note that for the experiments over the larger rt and rd data, we disregarded templates for which there is only one instance in the training data.

Original: This was shown to be beneficial in (Seidl et al., 2022), and we observed this as well.
Revised: This was shown to be beneficial in (Seidl et al., 2022), and we observed this as well.

Original: Single Step Models - Influence of Beam Size.
Revised: Single Step Models - Influence of Beam Size.

Original: We observed some variation in performance withchanged beam sizes.
Revised: We observed some variation in performance with changed beam sizes.

Original: However, since we could not identify a specific best value for the models - andeven not for individual models - we mostly resorted to the settings suggested by the authors; wemade an exception with G2S setting n_best= topk*3 since we experimented with larger topk than(Tu & Coley, 2021), which used a fix value.
Revised: However, since we could not identify a specific best value for the models - and even not for individual models - we mostly resorted to the settings suggested by the authors; we made an exception with G2S setting n_best=topk*3 since we experimented with larger topk than (Tu & Coley, 2021), which used a fix value.

Original: However, we found that since we also evaluated top-50,while some of the works considered maximally top-10.
Revised: However, we found that since we also evaluated top-50, while some of the works considered maximally top-10.

Original: We note that this also explains that there isno performance increase for GR in some cases from top-10 to top-50 (e.g., Table 5), but we obtainedoverall worse results for larger beam sizes and therefore used the ones originally proposed, keepingin mind that top-10 performance is likely more relevant in practice.
Revised: We note that this also explains that there is no performance increase for GR in some cases from top-10 to top-50 (e.g., Table 5), but we obtained overall worse results for larger beam sizes and therefore used the ones originally proposed, keeping in mind that top-10 performance is likely more relevant in practice.

Original: In particular, note that ourmulti-step experiments focus on maximally 10 predictions, hence this setting is in line with theones from the original studies.
Revised: In particular, note that our multi-step experiments focus on maximally 10 predictions, hence this setting is in line with the ones from the original studies.

Original: Multi-Step Experiments.
Revised: Multi-Step Experiments.

Original: We used the default settings suggested in AIZynthFinder (seehttps://molecularai.github.io/aizynthfinder/configuration.html , accessed 08/23/2022) with the following three exceptions.
Revised: We used the default settings suggested in AIZynthFinder (see https://molecularai.github.io/aizynthfinder/configuration.html , accessed 08/23/2022) with the following three exceptions.

Original: We used Retro ∗ as search algorithmsince we obtained extremely bad results with MCTS, probably because it uses the probabilitiesreturned by the models in a way that does not fit their current, mostly unregularized scale.
Revised: We used Retro ∗ as search algorithm since we obtained extremely bad results with MCTS, probably because it uses the probabilities returned by the models in a way that does not fit their current, mostly unregularized scale.

Original: Thesuggested maximal branching (cutoff_number) of 50 resulted in infeasible runtimes with mostnon-template-based models, hence we chose 10.
Revised: The suggested maximal branching (cutoff_number) of 50 resulted in infeasible runtimes with most non-template-based models, hence we chose 10.

Original: We note that this was also due to our filteringof invalid molecule predictions.
Revised: We note that this was also due to our filtering of invalid molecule predictions.

Original: Since the numbers of the latter were extremely high with, forexample, for CF, obtaining 50 valid reactant sets takes extremely long.
Revised: Since the numbers of the latter were extremely high with, for example, for CF, obtaining 50 valid reactant sets takes extremely long.

Original: Lastly, we increased themaximal number of reactions considered (max_transforms) to 10, which matches the maximalnumber in our data.
Revised: Lastly, we increased the maximal number of reactions considered (max_transforms) to 10, which matches the maximal number in our data.

Original: These adaptations show the need for considering practical aspects in modeldesign and evaluation.
Revised: These adaptations show the need for considering practical aspects in model design and evaluation.

Original: C About USPTO-50k and USPTO-full
Revised: B About USPTO-50k and USPTO-full

Original: Datasets.
Revised: Datasets.

Original: The two most commonly used datasets are USPTO-50k (Schneider et al., 2016) and
Revised: The two most commonly used datasets are USPTO-50k (Schneider et al., 2016) and

Original: USPTO-full (Dai et al., 2019), both subsets of the US patent data (Lowe, 2017).
Revised: USPTO-full (Dai et al., 2019), both subsets of the US patent data (Lowe, 2017).

Original: As the namessuggest, the latter, containing about 1M reactions, is much larger than the former; it is also used inmuch less works.
Revised: As the names suggest, the latter, containing about 1M reactions, is much larger than the former; it is also used in much less works.

Original: It has been observed recently, after our experimentation, that there are a fewnon-sensical data points in USPTO-50k (Lin et al., 2022); however, the split we used is still usefulin that it eases comparison to approaches we did not include in our study since it is one of onesmost commonly used.
Revised: It has been observed recently, after our experimentation, that there are a few non-sensical data points in USPTO-50k (Lin et al., 2022); however, the split we used is still useful in that it eases comparison to approaches we did not include in our study since it is one of ones most commonly used.

Original: We observed overlaps between the train and test data in USPTO-full andtherefore did not experiment over this dataset.
Revised: We observed overlaps between the train and test data in USPTO-full and therefore did not experiment over this dataset.

Original: Results, Tables 4, 5.
Revised: Results, Tables 4, 5.

Original: The table gives an overview of what is known about the SOTA.
Revised: The table gives an overview of what is known about the SOTA.

Original: Our results over
Revised: Our results over

Original: USPTO-50k (over a common split) are similar, see Table 5.
Revised: USPTO-50k (over a common split) are similar, see Table 5.

Original: Our additional analysis, presented inthis paper, completes the picture in terms of scalability and other aspects, such as generalization.
Revised: Our additional analysis, presented in this paper, completes the picture in terms of scalability and other aspects, such as generalization.

Original: Since not all models were originally evaluated over the same split (i.e., NPP, MHN, and CF(L)),we trained those.
Revised: Since not all models were originally evaluated over the same split (i.e., NPP, MHN, and CF(L)), we trained those.

Original: We re-evaluated all since this experiment served also to verify our wrapperimplementations.
Revised: We re-evaluated all since this experiment served also to verify our wrapper implementations.

Original: MHN vs. NeuralSym, Table 6.
Revised: MHN vs. NeuralSym, Table 6.

Original: Not surprisingly, the more complex MHN outperforms NeuralSym.
Revised: Not surprisingly, the more complex MHN outperforms NeuralSym.

Original: However, the comparison between these models is not entirely fair since the settings are quitedifferent.
Revised: However, the comparison between these models is not entirely fair since the settings are quite different.

Original: While NeuralSym uses rather standard features, MHN applies a 30k-dimensional vectorof different fingerprints and additionally uses the validation data during training.
Revised: While NeuralSym uses rather standard features, MHN applies a 30k-dimensional vector of different fingerprints and additionally uses the validation data during training.

Original: In order toobtain a better estimate of the actual impact of MHN’s main feature, the template encoding,we therefore consider two other models, NeuralSym+ (NeuralSym + MHN’s initial features) and
Revised: In order to obtain a better estimate of the actual impact of MHN’s main feature, the template encoding, we therefore consider two other models, NeuralSym+ (NeuralSym + MHN’s initial features) and

Original: NeuralSym++ (NPP) (NeuralSym+ trained including the validation data).
Revised: NeuralSym++ (NPP) (NeuralSym+ trained including the validation data).

Original: And we see that the gapindeed shrinks.
Revised: And we see that the gap indeed shrinks.

Original: [Table]
Revised: [Table]

Original: D Details about USPTO-ms
Revised: C Details about USPTO-ms

Original: We used the raw data from (Dai et al., 2019) and basically followed their data cleaning steps.
Revised: We used the raw data from (Dai et al., 2019) and basically followed their data cleaning steps.

Original: In anutshell, we then collected all products for which there are different reactions - after droppingreagents - , and dropped those reactions that are contained in the train/valid data of USPTO-50k(w.r.t.
Revised: In a nutshell, we then collected all products for which there are different reactions - after dropping reagents - , and dropped those reactions that are contained in the train/valid data of USPTO-50k (w.r.t.

Original: the spit of (Coley et al., 2017; Dai et al., 2019)).
Revised: the spit of (Coley et al., 2017; Dai et al., 2019)).

Original: The exact steps are documented in a notebookin our Git repository.
Revised: The exact steps are documented in a notebook in our Git repository.

Original: E Details about rt & rd
Revised: D Details about rt & rd

Original: Datasets, Table 7, Figures 7, 8 We basically followed the procedure proposed by (Mo et al., 2021;
Revised: Datasets, Table 7, Figures 7, 8 We basically followed the procedure proposed by (Mo et al., 2021;

Original: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.
Revised: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.

Original: The latterwas obtained from (Thakkar et al., 2020), who provide a thoroughly cleaned dataset containing areaction template with each reaction.
Revised: The latter was obtained from (Thakkar et al., 2020), who provide a thoroughly cleaned dataset containing a reaction template with each reaction.

Original: We note that, in this way, a filtering was done already (i.e.,all reactions where the template extraction failed), yet we considered the template availability asvaluable enough, especially for later analysis.
Revised: We note that, in this way, a filtering was done already (i.e., all reactions where the template extraction failed), yet we considered the template availability as valuable enough, especially for later analysis.

Original: However, we did not drop rare reactions (e.g., whosetemplate appears less often in the data) and adapted the last step, in which we created two customsplits.
Revised: However, we did not drop rare reactions (e.g., whose template appears less often in the data) and adapted the last step, in which we created two custom splits.

Original: Our time split is based on the year data coming with the reactions.
Revised: Our time split is based on the year data coming with the reactions.

Original: For the split containingespecially diverse routes as test and validation data we proceeded as suggested in (Genheden &
Revised: For the split containing especially diverse routes as test and validation data we proceeded as suggested in (Genheden &

Original: Bjerrum, 2022).
Revised: Bjerrum, 2022).

Original: F Details about n1 & n 
Revised: E Details about n1 & n

Original: Datasets, Table 8, Figures 7, 9.
Revised: Datasets, Table 8, Figures 7, 9.

Original: In order to see how general our data and results are, and to compareto the results from (Genheden & Bjerrum, 2022), we also experimented on their n1 and n5 datasets.
Revised: In order to see how general our data and results are, and to compare to the results from (Genheden & Bjerrum, 2022), we also experimented on their n1 and n5 datasets.

Original: Specifically, we extracted random test sets, of 1k and 5k routes for both n1 and n5, and evaluated [Figure] #Molecules L i k e l i h
Revised: Specifically, we extracted random test sets, of 1k and 5k routes for both n1 and n5, and evaluated our models trained over USPTO-50k over those.

Original: oo d        100.

Original: #Leaves train validtest rt-1k        100.

Original: #Reactions

Original: L i k e l i h

Original: oo d        100.

Original: #Longestlinearroute

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: l i

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: train valid test rd-1k

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: our models trained over USPTO-50k over those.

Original: Note that we created these datasets in the sameway as our rt and rd datasets, collecting all reactions contained in the considered routes.
Revised: Note that we created these datasets in the same way as our rt and rd datasets, collecting all reactions contained in the considered routes.

Original: Table 8shows the numbers of reactions, and Figure 9 shows the shares of reactions that match templatesfrom the USPTO-50k training data.
Revised: Tableshows the numbers of reactions, and Figure 9 shows the shares of reactions that match templates from the USPTO-50k training data.

Original: In this paper, we report the single-step results only for n1and n5.
Revised: In this paper, we report the single-step results only for n and n5.

Original: We did not observe greater differences to the results over our datasets in the multi-stepscenario and therefore did not complete the experiments, also due to the enormous resourcerequirements.
Revised: We did not observe greater differences to the results over our datasets in the multi-step scenario and therefore did not complete the experiments, also due to the enormous resource requirements.

Original: G Measuring Effectiveness: The Multi-Step Score (mss)
Revised: F Measuring Effectiveness: The Multi-Step Score (mss)

Original: Our multi-step score intends to complement existing metrics such as top-k accuracy by considering the route context.
Revised: Our multi-step score intends to complement existing metrics such as top-k accuracy by considering the route context.

Original: It is straightforward to calculate and interpret in that it captures the[Table] [Table] multi-step performance under the assumption that the model is combined with a perfect searchalgorithm.
Revised: It is straightforward to calculate and interpret in that it captures the multi-step performance under the assumption that the model is combined with a perfect search algorithm.

Original: Formally, let m k denote a single-step model which, given a target molecule, predicts aset of at most k sets of reactant molecules.
Revised: Formally, let m k denote a single-step model which, given a target molecule, predicts a set of at most k sets of reactant molecules.

Original: For a given tree structure representing a retrosyntheticroute, with nodes representing reactions, let S denote the set of all those reactions and S
Revised: For a given tree structure representing a retrosynthetic route, with nodes representing reactions, let S denote the set of all those reactions and S

Original: i denoteall reactions at level i 
Revised: i denote all reactions at level i (e

Original: (e.g., S 0 represents the singleton set containing the first reaction step at theroot node),
Revised: [Figure] g., S 0 represents the singleton set containing the first reaction step at the [Figure] #Molecules

Revised: L i k e l i h

Revised: oo d        100.

Revised: #Leaves train validtest rt-1k        100.

Original: such that S = (cid:83) i S i .
Revised: #Reactions L i k e l i h oo d        100. #Longestlinearroute      100.

Revised: #Molecules

Revised: L i k e l i h

Revised: oo d        100.

Revised: #Leaves train valid test rd-1k        100.

Original: Then we compute mss as follows:{+[Figure] #Reactions L i k e l i h oo d        
+}
Revised: {+[Figure] #Reactions L i k e l i h oo d        
+}

Original: T 0 :
Revised: [Figure] [Figure]

Original: = (cid:59)
Revised: [Figure]

Original: T 1 : = (cid:40) R, if S0 = { R»P }, R ∈ m k
Revised: root node), such that S = (cid:83) i S i . Then we compute mss as follows: [Equation]

Original: (P) (cid:59) , otherwise
Revised: (P) (cid:59) , otherwise

Original: T i + 1 : = T i ∪ (cid:40)(cid:83) R»P ∈ S i R,
Revised: T i + 1 : = T i ∪ (cid:40)(cid:83) R»P ∈ S i R,

Original: if ∀ R»P ∈ S i : R ∈ m k (P) and T i − 1 ̸= T i (cid:59) , otherwise mss 
Revised: if ∀ R»P ∈ S i : R ∈ m k (P) and T i − 1 ̸= T i (cid:59) , otherwise mss k ( m ,S 0 ,S 1 ,...) : = | T

Original: [Equation]
Revised: [Table]

Original: ( m ,S 0 ,S 1 ,...) : = | T ∞ | | S | 
Revised: | | S | [Table] [Table]

Original:  Note that in our evaluation, we consider a form of micro average, considering the predictions forall routes in the test set at once, and compute roughly (cid:80) T | T ∞ | (cid:80) S | S | .
Revised: [Table] [Table] [Table] Note that in our evaluation, we consider a form of micro average, considering the predictions for all routes in the test set at once, and compute roughly (cid:80) T | T ∞ | (cid:80) S | S | .

Original: For ease of computation, we alsoregard all nodes at one level at once and do not differentiate branches.
Revised: For ease of computation, we also regard all nodes at one level at once and do not differentiate branches.

Original: This processing can beconsidered in line with an interactive setting, which explores the tree breadth-first, one level ata time.[Figure] NoTemplate [Table]
Revised: This processing can be considered in line with an interactive setting, which explores the tree breadth-first, one level at a time.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: H Additional Results
Revised: G Additional Results

Original: Results, Tables 9, 10, 11, 12, Figure 10.
Revised: Results, Tables 9, 10, 11, 12, Figure 10.

Original: We extracted random subsets of n1 and n5 of two sizes,1k and 5k routes, to see how the results over the 1k sets generalize.
Revised: We extracted random subsets of n1 and n5 of two sizes, 1k and 5k routes, to see how the results over the 1k sets generalize.

Original: The tables show that thereare no considerable differences for all models.
Revised: The tables show that there are no considerable differences for all models.

Original: Figure 10 shows that all models struggle on thenew reactions; in particular, there are no great differences between top-1 and top-10, the latterare only slightly larger.
Revised: Figure 10 shows that all models struggle on the new reactions; in particular, there are no great differences between top-1 and top-10, the latter are only slightly larger.

Original: Tables 11 and 12 are discussed in the main paper.
Revised: Tables 11 and 12 are discussed in the main paper.

Original: For GR, we obtainedvery disappointing numbers.
Revised: For GR, we obtained very disappointing numbers.

Original: For verification purposes, we therefore also ran the GR versiontrained over USPTO-50k and this returned much better results; a possible reason might be thatthe vocabularies constructed based on our larger training sets are too large and confuse the model(while the vocabulary size for USPTO-50k is 174, we have 289 for rt and 307 for rd), so it might bepossible to obtain better results by dropping less common structures.
Revised: For verification purposes, we therefore also ran the GR version trained over USPTO-50k and this returned much better results; a possible reason might be that the vocabularies constructed based on our larger training sets are too large and confuse the model (while the vocabulary size for USPTO-50k is 174, we have 289 for rt and 307 for rd), so it might be possible to obtain better results by dropping less common structures.

Original: Also note that we set a traintime limit of 24 hours, as it was done in (Irwin et al., 2022).
Revised: Also note that we set a train time limit of 24 hours, as it was done in (Irwin et al., 2022).

Original: In particular NPP and CF might benefitfrom increasing that.
Revised: In particular NPP and CF might benefit from increasing that.

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: I Further Discussion
Revised: H Further Discussion

Original: • We show that the templates from the train data reveal interesting insights and alsoobserved a slight correlation between performance on reactions matching training templates and the number of template instances in the training data for all models (15-25%
Revised: • We show that the templates from the train data reveal interesting insights and also observed a slight correlation between performance on reactions matching training templates and the number of template instances in the training data for all models (15-25%

Original: Pearson correlation, depending on the model).
Revised: Pearson correlation, depending on the model).

Original: We also checked for a correlation between molecule complexity and prediction performance but did not observe any in ourexperiments.
Revised: We also checked for a correlation between molecule complexity and prediction performance but did not observe any in our experiments.

Original: Reaction complexity might be a factor worth to study in future work.•
Revised: Reaction complexity might be a factor worth to study in future work.

Original:  One problem with the non-template-based models is that they often produce invalidmolecular structures or reactions.
Revised: • One problem with the non-template-based models is that they often produce invalid molecular structures or reactions.

Original: While the former is technically no problem becausethose can be easily identified with tools such as RDKit, such a filtering step may yieldlarge runtime overhead.
Revised: While the former is technically no problem because those can be easily identified with tools such as RDKit, such a filtering step may yield large runtime overhead.

Original: Moreover, there is few efficient technology for identifying invalidreactions.
Revised: Moreover, there is few efficient technology for identifying invalid reactions.

Original: In order to get an idea of the validity of the predicted reactions, we brieflyexperimented with forward reaction prediction models as suggested in (Schwaller et al.,2020) and atom mapping tools for post processing in the multi-step experiments, but thiswas extremely time consuming as well.
Revised: In order to get an idea of the validity of the predicted reactions, we briefly experimented with forward reaction prediction models as suggested in (Schwaller et al., 2020) and atom mapping tools for post processing in the multi-step experiments, but this was extremely time consuming as well.

Original: Given this trade-off, it is likely that non-templatebased models will only be competitive if they either optimize for validity (amongst others)or include efficient post processing steps, to provide more efficient models; or if theyshow explicit benefits over template-based models, justifying the increase in runtime.• While the trends in the paper tend to show that the template-based models are mosteffective to date, we emphasize that they are limited by nature and will never allow for generalization.
Revised: Given this trade-off, it is likely that non-templatebased models will only be competitive if they either optimize for validity (amongst others) or include efficient post processing steps, to provide more efficient models; or if they show explicit benefits over template-based models, justifying the increase in runtime. • While the trends in the paper tend to show that the template-based models are most effective to date, we emphasize that they are limited by nature and will never allow for generalization.

Original: Although the current models do not seem to solve this task satisfactory yet, wesee some generalization capability.
Revised: Although the current models do not seem to solve this task satisfactory yet, we see some generalization capability.

Original: It will be interesting to see, how much generalizationis possible with state-of-the-art ML technology once this aspect is considered explicitly infuture studies.
Revised: It will be interesting to see, how much generalization is possible with state-of-the-art ML technology once this aspect is considered explicitly in future studies.

Original: We also note that there has been related work focusing on the generalization capability of the transformer across chemical space showing promising results (Leeet al., 2019); our study complements this type of investigation in pointing out reactiontemplates as efficient means to quantify generalization, even for non-template-basedmodels.
Revised: We also note that there has been related work focusing on the generalization capability of the transformer across chemical space showing promising results (Lee et al., 2019); our study complements this type of investigation in pointing out reaction templates as efficient means to quantify generalization, even for non-template-based models.

Original: J Limitations
Revised: I Limitations

Original: Our study focuses on more general aspects of evaluating retrosynthesis prediction and, in thecourse of this, provides an exemplary evaluation.
Revised: Our study focuses on more general aspects of evaluating retrosynthesis prediction and, in the course of this, provides an exemplary evaluation.

Original: As it is in the nature of experimental studies,there are various other possible and interesting settings which can and sometimes definitelyshould be explored in future work.
Revised: As it is in the nature of experimental studies, there are various other possible and interesting settings which can and sometimes definitely should be explored in future work.

Original: Below, we point out aspects which should be kept in mindwhen interpreting our results.
Revised: Below, we point out aspects which should be kept in mind when interpreting our results.

Original: • It has been noted in the past that the USPTO data is lacking certain kinds of reactions and is biased towards certain kinds of reaction (Schneider et al., 2016), hence it may not provide sufficient samples to train ML models useful for practice.
Revised: • It has been noted in the past that the USPTO data is lacking certain kinds of reactions and is biased towards certain kinds of reaction (Schneider et al., 2016), hence it may not provide sufficient samples to train ML models useful for practice.

Original: While this is likely the case, our study shows that the state of the art still struggles on the available data, in particular, in terms of generalization.
Revised: While this is likely the case, our study shows that the state of the art still struggles on the available data, in particular, in terms of generalization.

Original: Hence, the USPTO data seems to provide enoughchallenges for ML for the current moment.
Revised: Hence, the USPTO data seems to provide enough challenges for ML for the current moment.

Original: • We considered several template-based models but only two different template-free ones, and GraphRetro, while there are many others for each model class.
Revised: • We considered several template-based models but only two different template-free ones, and GraphRetro, while there are many others for each model class.

Original: It will be interesting to see if our observations regarding these representatives are similar for other models infuture studies.
Revised: It will be interesting to see if our observations regarding these representatives are similar for other models in future studies.

Original: • For most experiments, we used the models trained over USPTO-50k and the (hy- per)parameter settings suggested in the original studies, since benchmarking the SOTAwas out of the scope of this study.
Revised: • For most experiments, we used the models trained over USPTO-50k and the (hyper)parameter settings suggested in the original studies, since benchmarking the SOTA was out of the scope of this study.

Original: We experimented with some parameters, such asbeam sizes, and obtained slight variations in the results; however, for the parameters we considered, these variations did not lead to considerable changes in the overall trends.
Revised: We experimented with some parameters, such as beam sizes, and obtained slight variations in the results; however, for the parameters we considered, these variations did not lead to considerable changes in the overall trends.

Original: • In the multi-step experiments we fixed the settings using mostly the default settings from AIZynthFinder and Retro ∗ as search algorithm.
Revised: • In the multi-step experiments we fixed the settings using mostly the default settings from AIZynthFinder and Retro ∗ as search algorithm.

Original: We also resorted to creating the stocks representing the buyable molecules as suggested in (Genheden & Bjerrum, 2022).
Revised: We also resorted to creating the stocks representing the buyable molecules as suggested in (Genheden & Bjerrum, 2022).

Original: Certainly, variation in all these settings would lead to changes in the results.
Revised: Certainly, variation in all these settings would lead to changes in the results.

Original: Nevertheless, our evaluation often shows some considerable and more general differences between the models, which will likely generalize to some extent.
Revised: Nevertheless, our evaluation often shows some considerable and more general differences between the models, which will likely generalize to some extent.