Original: Retrosynthesis Prediction Revisited
Revised: Retrosynthesis Prediction Revisited

Original: Abstract
Revised: Abstract

Original: Retrosynthesis is an important problem in chemistry and represents an interesting challenge for AI since it involves predictions over sets of complex, molecular graph structures.
Revised: Retrosynthesis is an important problem in chemistry and represents an interesting challenge for AI since it involves predictions over sets of complex, molecular graph structures.

Original: Recently, a wealth of models ranging from language models to graph neural networks are being proposed.
Revised: Recently, a wealth of models ranging from language models to graph neural networks are being proposed.

Original: However, most studies evaluate over a single dataset and split only, focus on top-1 accuracy, and provide few insight into the actual capabilities of individual models.
Revised: However, most studies evaluate over a single dataset and split only, focus on top-1 accuracy, and provide few insight into the actual capabilities of individual models.

Original: This prevents research from moving forward since issues to be addressed by future work are not identified.
Revised: This prevents research from moving forward since issues to be addressed by future work are not identified.

Original: In this paper, we focus on the evaluation: we show that the currently used data does not fit to test generalization, one of the main goals stated in the literature; propose new splits of the USPTO reactions modeling various scenarios; study representatives of the main types of models over this data; and finally present the, to the best of our knowledge, first evaluation and comparison of these models in the multi-step scenario.
Revised: In this paper, we focus on the evaluation: we show that the currently used data does not fit to test generalization, one of the main goals stated in the literature; propose new splits of the USPTO reactions modeling various scenarios; study representatives of the main types of models over this data; and finally present the, to the best of our knowledge, first evaluation and comparison of these models in the multi-step scenario.

Original: Altogether, we show that the picture is more diverse than the results over the usually used USPTO-50k data suggest.
Revised: Altogether, we show that the picture is more diverse than the results over the usually used USPTO-50k data suggest.

Original:  Introduction
Revised:  Introduction

Original: Retrosynthesis is an important problem in chemistry (Robinson, 1917; Corey & Wipke, 1969): to synthesize a complex molecule in the lab based on purchasable molecules, a chemist needs to know the sequence of reactions to run.
Revised: Retrosynthesis is an important problem in chemistry (Robinson, 1917; Corey & Wipke, 1969): to synthesize a complex molecule in the lab based on purchasable molecules, a chemist needs to know the sequence of reactions to run.

Original: The problem represents an interesting challenge for AI since it involves predictions over molecular graph structures as well as a search component.
Revised: The problem represents an interesting challenge for AI since it involves predictions over molecular graph structures as well as a search component.

Original: In every step, a given product molecule (also, target ) is split into a set of reactant molecules , the targets for the next steps.
Revised: In every step, a given product molecule (also, target ) is split into a set of reactant molecules , the targets for the next steps.

Original: 2 Together, the steps form a retrosynthetic route , a tree structure.
Revised: 2 Together, the steps form a retrosynthetic route , a tree structure.

Original: The large number of models proposed for retrosynthesis prediction recently reflects the interest of the ML research community, for a good overview of the state of the art, we refer to the discussions in recent works (Tu & Coley, 2021; Zhong et al., 2022).
Revised: The large number of models proposed for retrosynthesis prediction recently reflects the interest of the ML research community, for a good overview of the state of the art, we refer to the discussions in recent works (Tu & Coley, 2021; Zhong et al., 2022).

Original: The single-step models used for predicting the individual steps can be categorized into three types: (I) Template-based approaches (Segler &
Revised: The single-step models used for predicting the individual steps can be categorized into three types: (I) Template-based approaches (Segler &

Original: Waller, 2017) classify a target w.r.t.
Revised: Waller, 2017) classify a target w.r.t.

Original: a set of reaction templates (i.e., rules how a target can be split if it meets certain conditions), which are usually mined from the training data or manually encoded, and assumed to encode valid reactions.
Revised: a set of reaction templates (i.e., rules how a target can be split if it meets certain conditions), which are usually mined from the training data or manually encoded, and assumed to encode valid reactions.

Original: (II) Template-free approaches (Liu et al., 2017) operate on the data alone and usually consider the molecules in text-based SMILES representation and the problem as a translation task (i.e., a sequence to sequence problem).
Revised: (II) Template-free approaches (Liu et al., 2017) operate on the data alone and usually consider the molecules in text-based SMILES representation and the problem as a translation task (i.e., a sequence to sequence problem).

Original: (III) Semi-template based approaches (Yan et al., 2020) apply a given atom mapping (i.e., considered as a kind of template), relating the atoms in the targets to the corresponding ones in the reactants, during training.
Revised: (III) Semi-template based approaches (Yan et al., 2020) apply a given atom mapping (i.e., considered as a kind of template), relating the atoms in the targets to the corresponding ones in the reactants, during training.

Original: Current approaches in this category model a two-step procedure first identifying the reaction center (i.e., where to split), and then completing the obtained molecule parts to reactants.
Revised: Current approaches in this category model a two-step procedure first identifying the reaction center (i.e., where to split), and then completing the obtained molecule parts to reactants.

Original: There are a few open-source tools for multi-step synthesis planning (Genheden et al., 2020; ask;
Revised: There are a few open-source tools for multi-step synthesis planning (Genheden et al., 2020; ask;

Original: Chen et al., 2020).
Revised: Chen et al., 2020).

Original: They apply a template-based MLP for the single steps by default, and resort to
Revised: They apply a template-based MLP for the single steps by default, and resort to

Original: Retro ∗ (Chen et al., 2020) (i.e., a variant of A ∗ search) or MCTS for the multi-step planning.
Revised: Retro ∗ (Chen et al., 2020) (i.e., a variant of A ∗ search) or MCTS for the multi-step planning.

Original: Studies about single-step models primarily compare on the USPTO-50k dataset (Schneider et al., 2016), since many works report numbers only for that data, and a single (often random) split.
Revised: Studies about single-step models primarily compare on the USPTO-50k dataset (Schneider et al., 2016), since many works report numbers only for that data, and a single (often random) split.

Original: This dataset was extracted from the United States patent data from 1976 to 2016 (Lowe, 2017).
Revised: This dataset was extracted from the United States patent data from 1976 to 2016 (Lowe, 2017).

Original: While the latter represents one of the few large open sources for reaction data, we note that most reactions (about 94%) in the USPTO-50k test set match templates one can extract from the training data and hence do not truly fit to test generalization.
Revised: While the latter represents one of the few large open sources for reaction data, we note that most reactions (about 94%) in the USPTO-50k test set match templates one can extract from the training data and hence do not truly fit to test generalization.

Original: Yet many of the template-free proposals consider the ability to generalize as main advantage of their approaches (Liu et al., 2017; Somnath et al., 2021).
Revised: Yet many of the template-free proposals consider the ability to generalize as main advantage of their approaches (Liu et al., 2017; Somnath et al., 2021).

Original: Moreover, there is a strong focus on improving top-1 accuracy, which has been critisized as being insufficient in the chemistry community (Schwaller et al., 2019).
Revised: Moreover, there is a strong focus on improving top-1 accuracy, which has been critisized as being insufficient in the chemistry community (Schwaller et al., 2019).

Original: Generally, few works provide insight into the actual capabilities of individual models and components, and concrete comparisons between models (i.e., in terms of predictions) are rare, possibly because many implementations are less accessible.
Revised: Generally, few works provide insight into the actual capabilities of individual models and components, and concrete comparisons between models (i.e., in terms of predictions) are rare, possibly because many implementations are less accessible.

Original: The latter also often ignore practical aspects, such that applying the models out of the box would be hard.
Revised: The latter also often ignore practical aspects, such that applying the models out of the box would be hard.

Original: For example, most language-based models operate in batch mode reading and writing from files (Zheng et al., 2019; Chen et al., 2019), and others require involved preprocessing or training procedures (Dai et al., 2019; Yan et al., 2020).
Revised: For example, most language-based models operate in batch mode reading and writing from files (Zheng et al., 2019; Chen et al., 2019), and others require involved preprocessing or training procedures (Dai et al., 2019; Yan et al., 2020).

Original: Altogether, this prevents research from moving forward since issues that are critical for applicability are not identified, and research is rarely translated into applications.
Revised: Altogether, this prevents research from moving forward since issues that are critical for applicability are not identified, and research is rarely translated into applications.

Original: Related Work.
Revised: Related Work.

Original: While we above describe the general trend, there are certainly exceptions.
Revised: While we above describe the general trend, there are certainly exceptions.

Original: In particular, several analyses in the chemistry domain provide more detail, in terms of all aspects, data, methodology, and also in measuring effectiveness in the multi-step scenario.
Revised: In particular, several analyses in the chemistry domain provide more detail, in terms of all aspects, data, methodology, and also in measuring effectiveness in the multi-step scenario.

Original: Data.
Revised: Data.

Original: Some studies also evaluate over larger USPTO subsets (Dai et al., 2019) or use different splits (Seidl et al., 2022; Chen et al., 2019).
Revised: Some studies also evaluate over larger USPTO subsets (Dai et al., 2019) or use different splits (Seidl et al., 2022; Chen et al., 2019).

Original: Methodology.
Revised: Methodology.

Original: Recently, (Lin et al., 2022) proposed to re-rank predictions and, in the course of this, compared different single-step models and, specifically, also in terms of their actual predictions.
Revised: Recently, (Lin et al., 2022) proposed to re-rank predictions and, in the course of this, compared different single-step models and, specifically, also in terms of their actual predictions.

Original: Further, (Chen et al., 2019; Lin et al., 2022) distinguish the predictions in terms of reaction types.
Revised: Further, (Chen et al., 2019; Lin et al., 2022) distinguish the predictions in terms of reaction types.

Original: There have been further various proposals for metrics beyond top-k.
Revised: There have been further various proposals for metrics beyond top-k.

Original: (Schwaller et al., 2019, 2020) propose round-trip accuracy , the agreement between two models predicting the reaction in the forward and, respectively, backward direction; coverage , quantifying for how many of the products at least one valid suggestion of the set of reactants could be found; and diversity , the number of diverse, valid reactants after removing the buyable ones.
Revised: (Schwaller et al., 2019, 2020) propose round-trip accuracy , the agreement between two models predicting the reaction in the forward and, respectively, backward direction; coverage , quantifying for how many of the products at least one valid suggestion of the set of reactants could be found; and diversity , the number of diverse, valid reactants after removing the buyable ones.

Original: Regarding diversity, they further consider a metric of statistical significance.
Revised: Regarding diversity, they further consider a metric of statistical significance.

Original: (Tetko et al., 2020) propose to report MaxFrag , the recognition of the largest reactant, to estimate the ability of a model to deduce the correct reaction class; note the similarity to the coverage from (Schwaller et al., 2020).
Revised: (Tetko et al., 2020) propose to report MaxFrag , the recognition of the largest reactant, to estimate the ability of a model to deduce the correct reaction class; note the similarity to the coverage from (Schwaller et al., 2020).

Original: Most recently, (Lin et al., 2022) applied MRR , observing that it reflects a trend in between the ones of top1 and top-3 accuracy, and area under the top-k curve , which parallels top-k accuracy; while both are similar to the latter in trend, especially MRR offers better explainability in that it summarizes the latter for different k. Multi-step.
Revised: Most recently, (Lin et al., 2022) applied MRR , observing that it reflects a trend in between the ones of top1 and top-3 accuracy, and area under the top-k curve , which parallels top-k accuracy; while both are similar to the latter in trend, especially MRR offers better explainability in that it summarizes the latter for different k. Multi-step.

Original: Lastly, some works apply the proposed single-step model in the muti-step planning setting by either chaining single-step predictions (Coley et al., 2017) or combined with a search algorithm (Ishida et al., 2022) but, to the best of our knowledge, there are no comparisons between models.
Revised: Lastly, some works apply the proposed single-step model in the muti-step planning setting by either chaining single-step predictions (Coley et al., 2017) or combined with a search algorithm (Ishida et al., 2022) but, to the best of our knowledge, there are no comparisons between models.

Original: Such a comparison has come closer, with the recently proposed PARoutes framework (Genheden & Bjerrum, 2022), which offers tools for extracting synthetic routes from the USPTO patent data, based on the work of (Mo et al., 2021), and for comparing predicted routes to the extracted ones.
Revised: Such a comparison has come closer, with the recently proposed PARoutes framework (Genheden & Bjerrum, 2022), which offers tools for extracting synthetic routes from the USPTO patent data, based on the work of (Mo et al., 2021), and for comparing predicted routes to the extracted ones.

Original: (Genheden & Bjerrum, 2022) also propose two route datasets, n1 and n5, and present an evaluation using template-based single-step models.
Revised: (Genheden & Bjerrum, 2022) also propose two route datasets, n1 and n5, and present an evaluation using template-based single-step models.

Original: However, this route data was designed for evaluating model capabilities under rather controlled conditions, it does not contain rare reactions and the training data is in a certain sense artificial (exactly three reaction examples per reaction template).
Revised: However, this route data was designed for evaluating model capabilities under rather controlled conditions, it does not contain rare reactions and the training data is in a certain sense artificial (exactly three reaction examples per reaction template).

Original: Therefore, we used this framework for the creation of more general datasets, and in our evaluation of the multi-step setting.
Revised: Therefore, we used this framework for the creation of more general datasets, and in our evaluation of the multi-step setting.

Original: There are some other route datasets: (Chen et al., 2020) also extracted routes from the USPTO data that do not contain rare reactions, and (Ishida et al., 2022) manually curated routes.
Revised: There are some other route datasets: (Chen et al., 2020) also extracted routes from the USPTO data that do not contain rare reactions, and (Ishida et al., 2022) manually curated routes.

Original: In summary, there are many reasonable proposals to evaluate single-step retrosynthesis models, yet these have not been picked up by the broader community and are not commonly used today.
Revised: In summary, there are many reasonable proposals to evaluate single-step retrosynthesis models, yet these have not been picked up by the broader community and are not commonly used today.

Original: In the words of (Schwaller et al., 2019), the evaluation of single-step retrosynthetic models is an overlooked research topic .
Revised: In the words of (Schwaller et al., 2019), the evaluation of single-step retrosynthetic models is an overlooked research topic .

Original: Our study focuses on the evaluation of single-step models, applies some of these ideas, demonstrates their relevance for applications, and complements the above works by focusing on important aspects that have been less in focus so far, namely, prediction diversity, generalization, and performance in the multi-step scenario.
Revised: Our study focuses on the evaluation of single-step models, applies some of these ideas, demonstrates their relevance for applications, and complements the above works by focusing on important aspects that have been less in focus so far, namely, prediction diversity, generalization, and performance in the multi-step scenario.

Original: Our contributions are the following.
Revised: Our contributions are the following.

Original: • We propose new splits of the USPTO reactions.
Revised: • We propose new splits of the USPTO reactions.

Original: Since prediction diversity is important in view of the incomplete reaction data (e.g., missing conditions which may restrict applicability) and the models not being perfect, we collected products which can be synthesized from different reactant sets and thus have multiple solutions in the test set USPTO-ms .
Revised: Since prediction diversity is important in view of the incomplete reaction data (e.g., missing conditions which may restrict applicability) and the models not being perfect, we collected products which can be synthesized from different reactant sets and thus have multiple solutions in the test set USPTO-ms .

Original: Moreover, we create datasets USPTO-rt and USPTO-rd containing (the reactions of) retrosynthetic routes such that we can compare the performance in the single and multi-step scenarios by either evaluating single-step models on the reactions with all reactions’ targets as input, or by running them inside a synthesis planning tool with only the roots of the routes as input.
Revised: Moreover, we create datasets USPTO-rt and USPTO-rd containing (the reactions of) retrosynthetic routes such that we can compare the performance in the single and multi-step scenarios by either evaluating single-step models on the reactions with all reactions’ targets as input, or by running them inside a synthesis planning tool with only the roots of the routes as input.

Original: This comparison shows that there is indeed a discrepancy between the performance in the two scenarios.
Revised: This comparison shows that there is indeed a discrepancy between the performance in the two scenarios.

Original: • In addition, we develop the multi-step score (mss) for estimating the multi-step performance based on the predictions on individual reactions, independently of a route search algorithm.
Revised: • In addition, we develop the multi-step score (mss) for estimating the multi-step performance based on the predictions on individual reactions, independently of a route search algorithm.

Original: • We evaluate several representatives of the main models types over various datasets and present the, to the best of our knowledge, first multi-step evaluation and comparison of these models.
Revised: • We evaluate several representatives of the main models types over various datasets and present the, to the best of our knowledge, first multi-step evaluation and comparison of these models.

Original: • To address issue of missing comparisons, we created all our test sets such that they do not overlap with the USPTO-50k train and validation data (based on the usually used split from (Coley et al., 2017; Dai et al., 2019)).
Revised: • To address issue of missing comparisons, we created all our test sets such that they do not overlap with the USPTO-50k train and validation data (based on the usually used split from (Coley et al., 2017; Dai et al., 2019)).

Original: Hence, researchers can use the existing checkpoints and evaluate easily over our new test sets.
Revised: Hence, researchers can use the existing checkpoints and evaluate easily over our new test sets.

Original: For the evaluation, we created a simple, abstract wrapper class for singlestep retrosynthesis models; evaluation scripts based on this wrapper; and example wrappers for the models we evaluate in this work.
Revised: For the evaluation, we created a simple, abstract wrapper class for singlestep retrosynthesis models; evaluation scripts based on this wrapper; and example wrappers for the models we evaluate in this work.

Original: Furthermore, we provide an extension of AIZynthFinder (Genheden et al., 2020), such that it can be used directly with our wrapper interface.
Revised: Furthermore, we provide an extension of AIZynthFinder (Genheden et al., 2020), such that it can be used directly with our wrapper interface.

Original: All assets and code are provided at https://github.com/vthost/retroeval .
Revised: All assets and code are provided at https://github.com/vthost/retroeval .

Original: The
Revised: The

Original: fast
Revised: fast

Original: adoption of Retro
Revised: adoption of Retro

Original: ∗ (Genheden et al., 2020; Kim et al., 2021) has
Revised: ∗ (Genheden et al., 2020; Kim et al., 2021) has

Original: shown that the community is
Revised: shown that the community is

Original: open to adopt latest
Revised: open to adopt latest

Original: research
Revised: research

Original: outcomes if they are accessible,
Revised: outcomes if they are accessible,

Original: efficient,
Revised: efficient,

Original: and
Revised: and

Original: effective.
Revised: effective.

Original: The
Revised: The

Original: goal of
Revised: goal of

Original: our work is to
Revised: our work is to

Original: ease the development of single-
Revised: ease the development of single-

Original: step models towards this direction.
Revised: step models towards this direction.

Original:  Our Setting
Revised:  Our Setting

Original: Models considered in this paper.
Revised: Models considered in this paper.

Original: • Template-based:
Revised: • Template-based:

Original: NeuralSym (NPP) (Segler & Waller, 2017), a regular MLP; the Conditional Graph Logic Network (GLN) (Dai et al., 2019), which models template applicability using a conditional graphical model that is parameterized using a structure-based embedding; and the recently proposed MHNreact (MHN) (Seidl et al., 2022), which uses a modern Hopfield network (Widrich et al., 2020) to retrieve stored templates based on the input molecule embedding.
Revised: NeuralSym (NPP) (Segler & Waller, 2017), a regular MLP; the Conditional Graph Logic Network (GLN) (Dai et al., 2019), which models template applicability using a conditional graphical model that is parameterized using a structure-based embedding; and the recently proposed MHNreact (MHN) (Seidl et al., 2022), which uses a modern Hopfield network (Widrich et al., 2020) to retrieve stored templates based on the input molecule embedding.

Original: Both GLN and MHN also encode the templates and hence can exploit similarities between them.
Revised: Both GLN and MHN also encode the templates and hence can exploit similarities between them.

Original: • Template-free: Chemformer (CF) and Chemformer-Large (CFL) (Irwin et al., 2022), regular BART language models (Lewis et al., 2020) that were pre-trained on a large molecule corpus and only differ in size; and Graph2SMILES (G2S)
Revised: • Template-free: Chemformer (CF) and Chemformer-Large (CFL) (Irwin et al., 2022), regular BART language models (Lewis et al., 2020) that were pre-trained on a large molecule corpus and only differ in size; and Graph2SMILES (G2S)

Original: (Tu & Coley, 2021), combining a graph-based transformer encoder with a sequence-based decoder.
Revised: (Tu & Coley, 2021), combining a graph-based transformer encoder with a sequence-based decoder.

Original: • Semi-template-based:
Revised: • Semi-template-based:

Original: GraphRetro (GR) (Somnath et al., 2021), modeling intermediate reaction steps with GNNs.
Revised: GraphRetro (GR) (Somnath et al., 2021), modeling intermediate reaction steps with GNNs.

Original: Note that we tried to set up other models, especially from the last category, but often failed.
Revised: Note that we tried to set up other models, especially from the last category, but often failed.

Original: Further, we were able to run GLN only locally and therefore present less results for this model.
Revised: Further, we were able to run GLN only locally and therefore present less results for this model.

Original: Metrics.
Revised: Metrics.

Original: Top-k accuracy ( top-k ) represents the fraction of test targets for which the correct reactant set is among the top-k ranked predicted reactant sets.
Revised: Top-k accuracy ( top-k ) represents the fraction of test targets for which the correct reactant set is among the top-k ranked predicted reactant sets.

Original: Note that the former is usually used in a setting where a target with multiple correct reactant sets is considered as two different test points and the model is evaluated twice w.r.t.
Revised: Note that the former is usually used in a setting where a target with multiple correct reactant sets is considered as two different test points and the model is evaluated twice w.r.t.

Original: the two solutions, respectively.
Revised: the two solutions, respectively.

Original: We further consider the mean reciprocal rank ( MRR ) and MaxFrag accuracy at k ( mf-k ), focusing on the recognition of the largest reactant.
Revised: We further consider the mean reciprocal rank ( MRR ) and MaxFrag accuracy at k ( mf-k ), focusing on the recognition of the largest reactant.

Original: In the scenario where we focus on predicting multiple solutions, we consider recall@k ( r@k ), which represents the proportion of correct reactant sets among the top-k ranked predictions averaged over all
Revised: In the scenario where we focus on predicting multiple solutions, we consider recall@k ( r@k ), which represents the proportion of correct reactant sets among the top-k ranked predictions averaged over all

Original: [Table] [Table]
Revised: [Table] [Table]

Original: # T a r g e t s
Revised: # T a r g e t s

Original: Reactions
Revised: Reactions

Original: [Table]
Revised: [Table]

Original: target
Revised: target

Original: TemplateOther
Revised: TemplateOther

Original: [Table]
Revised: [Table]

Original:  Revisiting USPTO-50k
Revised:  Revisiting USPTO-50k

Original: In most experiments, we consider the models trained over USPTO-50k (or short, 50k).
Revised: In most experiments, we consider the models trained over USPTO-50k (or short, 50k).

Original: If available, we used the checkpoints from the original studies within our wrappers.
Revised: If available, we used the checkpoints from the original studies within our wrappers.

Original: Some of the trends we observe turned out to be more general (see also Sec. 6 and Appendix B):
Revised: Some of the trends we observe turned out to be more general (see also Sec. 6 and Appendix B):

Original: • top-1: CF(L) is best, followed by (in no particular order) G2S and GR.
Revised: • top-1: CF(L) is best, followed by (in no particular order) G2S and GR.

Original: • MRR, top-5-50: Template-based models are best, followed by (in no particular order)
Revised: • MRR, top-5-50: Template-based models are best, followed by (in no particular order)

Original: G2S, GR.
Revised: G2S, GR.

Original: • G2S performs more similar to GR than to CF(L).
Revised: • G2S performs more similar to GR than to CF(L).

Original: CF-L usually outperforms CL slightly.
Revised: CF-L usually outperforms CL slightly.

Original: • top-50: GLN/MHN basically reach maximal
Revised: • top-50: GLN/MHN basically reach maximal

Original: possible
Revised: possible

Original: performance given the
Revised: performance given the

Original: amount
Revised: amount

Original: of test
Revised: of test

Original: reactions matching templates
Revised: reactions matching templates

Original: (see
Revised: (see

Original: Figure 4)
Revised: Figure 4)

Original: In particular
Revised: In particular

Original: they are already
Revised: they are already

Original: close to that
Revised: close to that

Original: at
Revised: at

Original: top-
Revised: top-

Original: Note that,
Revised: Note that,

Original: the template-
Revised: the template-

Original: based top-k predictions are selected from those templates that are actually applicable (i.e., their conditions are satisfied by the target) but our NeuralSym and MHN implementations pre-compute only the applicability of the first 200 templates; when considering all, MHN reaches 93.2% at top-50.
Revised: based top-k predictions are selected from those templates that are actually applicable (i.e., their conditions are satisfied by the target) but our NeuralSym and MHN implementations pre-compute only the applicability of the first 200 templates; when considering all, MHN reaches 93.2% at top-50.

Original: • MRR is between top-1 and top-5, and mf-10 is more similar to top-50 than to top-10.
Revised: • MRR is between top-1 and top-5, and mf-10 is more similar to top-50 than to top-10.

Original:  Prediction Diversity: The Dataset USPTO-ms
Revised:  Prediction Diversity: The Dataset USPTO-ms

Original: USPTO-ms.
Revised: USPTO-ms.

Original: For evaluating how well the models capture diversity, we collected reactions from the USPTO patent data (Lowe, 2017) where we have multiple solutions (i.e., multiple possible reactant sets) into the new test set USPTO-ms. Details about the creation process can be found in Appendix C.
Revised: For evaluating how well the models capture diversity, we collected reactions from the USPTO patent data (Lowe, 2017) where we have multiple solutions (i.e., multiple possible reactant sets) into the new test set USPTO-ms. Details about the creation process can be found in Appendix C.

Original: The data contains 3,501 different targets and 7,438 different reactions.
Revised: The data contains 3,501 different targets and 7,438 different reactions.

Original: Figureshows further statistics.
Revised: Figureshows further statistics.

Original: Note that we dropped reactions with more than six solutions since these were extremely rare.
Revised: Note that we dropped reactions with more than six solutions since these were extremely rare.

Original: We see that only about a quarter of the reactions match templates from the
Revised: We see that only about a quarter of the reactions match templates from the

Original: USPTO-50k train/valid data, hence it is not only the diversity but also the novelty of the reactions which represents a challenge for the models.
Revised: USPTO-50k train/valid data, hence it is not only the diversity but also the novelty of the reactions which represents a challenge for the models.

Original: It is possible that the latter is the greater challenge.
Revised: It is possible that the latter is the greater challenge.

Original: [Figure]
Revised: [Figure]

Original: NPP
Revised: NPP

Original: GLN
Revised: GLN

Original: MHN
Revised: MHN

Original: CFL
Revised: CFL

Original: G2S
Revised: G2S

Original: GR
Revised: GR

Original: Results, Table 1, Fig. 2.
Revised: Results, Table 1, Fig. 2.

Original: At first glance, the picture is very different from the one over USPTO-50k.
Revised: At first glance, the picture is very different from the one over USPTO-50k.

Original: In particular, we observe no considerable differences between the models at r@1, with the exception of
Revised: In particular, we observe no considerable differences between the models at r@1, with the exception of

Original: NPP.
Revised: NPP.

Original: Note that we include the scores on the test data to show the maximal r@k possible.
Revised: Note that we include the scores on the test data to show the maximal r@k possible.

Original: Further, good scores at k = 1..3 are especially hard to obtain because most targets have two or three solutions.
Revised: Further, good scores at k = 1..3 are especially hard to obtain because most targets have two or three solutions.

Original: At higher k’s it gets again easier for the models since they allow for mistakes.
Revised: At higher k’s it gets again easier for the models since they allow for mistakes.

Original: Here, the template-based models and G2S perform especially well.
Revised: Here, the template-based models and G2S perform especially well.

Original: Since the table shows the average across targets, Figure 2, additionally shows how many solutions the models actually find per target among the top 5.
Revised: Since the table shows the average across targets, Figure 2, additionally shows how many solutions the models actually find per target among the top 5.

Original: We see that the r@k performance can be largely attributed to single solutions.
Revised: We see that the r@k performance can be largely attributed to single solutions.

Original: In particular, CF(L) seems to lack diversity.
Revised: In particular, CF(L) seems to lack diversity.

Original: #Reactions
Revised: #Reactions

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        100.
Revised: oo d        100.

Original: #Longestlinearroute train validtest rt-1k        100.
Revised: #Longestlinearroute train validtest rt-1k        100.

Original: #Reactions
Revised: #Reactions

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        100.
Revised: oo d        100.

Original: #Longestlinearroute train validtest rd-1k
Revised: #Longestlinearroute train validtest rd-1k

Original: [Figure]
Revised: [Figure]

Original:  On Evaluating Single & Multi-Step Prediction
Revised:  On Evaluating Single & Multi-Step Prediction

Original: Prediction in Route Context, the Datasets rt & rd, Figure 3, Table 7.
Revised: Prediction in Route Context, the Datasets rt & rd, Figure 3, Table 7.

Original: We followed (Mo et al., 2021;
Revised: We followed (Mo et al., 2021;

Original: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.
Revised: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.

Original: However, we did not drop rare reactions and created two custom splits.
Revised: However, we did not drop rare reactions and created two custom splits.

Original: In particular, we created a time split, the dataset USPTO-rt (or short, rt ), and one where we extracted sets of very diverse routes as test and validation data and kept all remaining routes as train data, USPTO-rd (or short, rd ).
Revised: In particular, we created a time split, the dataset USPTO-rt (or short, rt ), and one where we extracted sets of very diverse routes as test and validation data and kept all remaining routes as train data, USPTO-rd (or short, rd ).

Original: Further details about the creation are given in Appendix D.
Revised: Further details about the creation are given in Appendix D.

Original: The figures show that the rt data contains train/valid/test distributions that are very similar, while there is great diversity in rd.
Revised: The figures show that the rt data contains train/valid/test distributions that are very similar, while there is great diversity in rd.

Original: The latter dataset contains more data overall because the test and validation routes are longer and we kept the data from all other patents for training.
Revised: The latter dataset contains more data overall because the test and validation routes are longer and we kept the data from all other patents for training.

Original: For rt we dropped all remaining data from patents at the test time points.
Revised: For rt we dropped all remaining data from patents at the test time points.

Original: We will use these full rt and rd datasets particularly to compare how the model performance changes when trained over data beyond USPTO-50k.
Revised: We will use these full rt and rd datasets particularly to compare how the model performance changes when trained over data beyond USPTO-50k.

Original: Test Subsets: rt-1k & rd-1k, Figure 7, Table 7.
Revised: Test Subsets: rt-1k & rd-1k, Figure 7, Table 7.

Original: Since the models turned out to be more resource and especially time-intensive than the regularly used MLPs in the multi-step scenario, we extracted subsets of the original rt and rd test sets which will be in the focus of our study.
Revised: Since the models turned out to be more resource and especially time-intensive than the regularly used MLPs in the multi-step scenario, we extracted subsets of the original rt and rd test sets which will be in the focus of our study.

Original: Each contains (the reactions of) 1k routes.
Revised: Each contains (the reactions of) 1k routes.

Original: We additionally ensured that all reactions are unique, that we have maximally one route per patent, and that there is no overlap with the USPTO-50k train/valid data.
Revised: We additionally ensured that all reactions are unique, that we have maximally one route per patent, and that there is no overlap with the USPTO-50k train/valid data.

Original: The overview of our test sets compared to the ones from (Genheden & Bjerrum, 2022) in the figure shows that the statistics are similar.
Revised: The overview of our test sets compared to the ones from (Genheden & Bjerrum, 2022) in the figure shows that the statistics are similar.

Original: For comparison, we also evaluated over subsets of n1 and n5, but did not observe considerable differences; hence the latter lie primarily in the nature of the training data (i.e., the n1/n5 train data is more similar to the test data).
Revised: For comparison, we also evaluated over subsets of n1 and n5, but did not observe considerable differences; hence the latter lie primarily in the nature of the training data (i.e., the n1/n5 train data is more similar to the test data).

Original: The Multi-Step Score (mss).
Revised: The Multi-Step Score (mss).

Original: In a nutshell, mss counts targets only then as solved if the targets occurring previously in the route are solved as well; see Appendix F for a formal definition.
Revised: In a nutshell, mss counts targets only then as solved if the targets occurring previously in the route are solved as well; see Appendix F for a formal definition.

Original: We will see that mss nicely complements existing metrics.
Revised: We will see that mss nicely complements existing metrics.

Original:  Evaluation in Route Context
Revised:  Evaluation in Route Context

Original: Single-Step Results on 1k Test Sets, Tables 2, Figure 5.
Revised: Single-Step Results on 1k Test Sets, Tables 2, Figure 5.

Original: Our evaluation over all test sets generally confirms the trends outlined in Section 3 in the context of USPTO50k, including
Revised: Our evaluation over all test sets generally confirms the trends outlined in Section 3 in the context of USPTO50k, including

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: the additional results on n1 and n5 in Appendix G.
Revised: the additional results on n1 and n5 in Appendix G.

Original: Still, there are notable differences.
Revised: Still, there are notable differences.

Original: The numbers are much lower, showing that the data from (Genheden & Bjerrum, 2022) as well as ours is more challenging.
Revised: The numbers are much lower, showing that the data from (Genheden & Bjerrum, 2022) as well as ours is more challenging.

Original: More interestingly, at top-1, the template-based model
Revised: More interestingly, at top-1, the template-based model

Original: MHN is basically en par with G2S and GR.
Revised: MHN is basically en par with G2S and GR.

Original: The Chemformers are clearly winning but considerably worse at higher k’s.
Revised: The Chemformers are clearly winning but considerably worse at higher k’s.

Original: The results on all datasets are in similar ranges with the ones for rd-1k being lowest.
Revised: The results on all datasets are in similar ranges with the ones for rd-1k being lowest.

Original: In
Revised: In

Original: Figure 5, we can see that the performance can be attributed primarily to the targets matching templates seen during training.
Revised: Figure 5, we can see that the performance can be attributed primarily to the targets matching templates seen during training.

Original: There are only few signals, although there are some, that the semi/non-templatebased models are capable to generalize, especially G2S and GR.
Revised: There are only few signals, although there are some, that the semi/non-templatebased models are capable to generalize, especially G2S and GR.

Original: Our mss score differs considerably in magnitude.
Revised: Our mss score differs considerably in magnitude.

Original: Interestingly, its trend is very similar to top-k for k > 1, although it is interpreted differently.
Revised: Interestingly, its trend is very similar to top-k for k > 1, although it is interpreted differently.

Original: Nevertheless note that it better reflects the route context, in that the numbers are lowest on the more complex routes (rd and n5) while, for example, top-5 on n5-1k is higher than on n1-1k; and the difference to mss on the routes in rt is larger than, for instance, the differences at MRR.
Revised: Nevertheless note that it better reflects the route context, in that the numbers are lowest on the more complex routes (rd and n5) while, for example, top-5 on n5-1k is higher than on n1-1k; and the difference to mss on the routes in rt is larger than, for instance, the differences at MRR.

Original: Comparison of Predictions, Figure 6.
Revised: Comparison of Predictions, Figure 6.

Original: We conclude the analysis of the single-step models trained on USPTO-50k by comparing the actual predictions, w.r.t.
Revised: We conclude the analysis of the single-step models trained on USPTO-50k by comparing the actual predictions, w.r.t.

Original: a top-5 threshold, in terms of the pairwise agreement between models on solutions, averaged over all six datasets, and the exact opposite, the percentage of solutions predicted by a single model only, w.r.t.
Revised: a top-5 threshold, in terms of the pairwise agreement between models on solutions, averaged over all six datasets, and the exact opposite, the percentage of solutions predicted by a single model only, w.r.t.

Original: the total number of those solutions per dataset.
Revised: the total number of those solutions per dataset.

Original: The former shows that there seems to be a correlation within a class of models.
Revised: The former shows that there seems to be a correlation within a class of models.

Original: While the absolute results show greater similarity between G2S and GR (i.e., instead of
Revised: While the absolute results show greater similarity between G2S and GR (i.e., instead of

Original: G2S and CF(L)), here it is slightly larger between G2S and CFL, also between the template-based models.
Revised: G2S and CF(L)), here it is slightly larger between G2S and CFL, also between the template-based models.

Original: Note that (Lin et al., 2022) provide a similar but more fine
Revised: Note that (Lin et al., 2022) provide a similar but more fine

Original: Scaling Up the Train Data, Tables 11, 12.
Revised: Scaling Up the Train Data, Tables 11, 12.

Original: On the 1k test sets we can compare to the models trained over USPTO-50k (see Table 2).
Revised: On the 1k test sets we can compare to the models trained over USPTO-50k (see Table 2).

Original: The impact of retraining turns out to be much higher for rd-1k than for rt-1k; for example, we have a top-1 increase of about 10%, vs. 3-5% on rt-1k.
Revised: The impact of retraining turns out to be much higher for rd-1k than for rt-1k; for example, we have a top-1 increase of about 10%, vs. 3-5% on rt-1k.

Original: This might be explained by the larger amount of training data in rd-1k and, in particular, by the greater similarity between train and test data.
Revised: This might be explained by the larger amount of training data in rd-1k and, in particular, by the greater similarity between train and test data.

Original: Recall that we have strict temporal split with rt.
Revised: Recall that we have strict temporal split with rt.

Original: One very likely explanation, and consequence of the similarity, is the increase in test reactions matching templates seen during training.
Revised: One very likely explanation, and consequence of the similarity, is the increase in test reactions matching templates seen during training.

Original: Figure 4, shows that the share for rd is now larger than the one for rt.
Revised: Figure 4, shows that the share for rd is now larger than the one for rt.

Original: Apparently, this impacts the performance of the template-free CF nearly as much as the template-based models.
Revised: Apparently, this impacts the performance of the template-free CF nearly as much as the template-based models.

Original: Finally, we note that there is no gap anymore between the latter; in fact, for larger k’s, which we consider in the multi-step setting, NPP seems to provide better performance.
Revised: Finally, we note that there is no gap anymore between the latter; in fact, for larger k’s, which we consider in the multi-step setting, NPP seems to provide better performance.

Original: The results on the full test sets are very similar and shown in Table 12.
Revised: The results on the full test sets are very similar and shown in Table 12.

Original: The Multi-Step Scenario, Table 3.
Revised: The Multi-Step Scenario, Table 3.

Original: Our results give insights into the models’ out-of-the-box usability.
Revised: Our results give insights into the models’ out-of-the-box usability.

Original: We show the top-5 accuracy w.r.t.
Revised: We show the top-5 accuracy w.r.t.

Original: finding the exact test route.
Revised: finding the exact test route.

Original: Since this criterion is rather strong, we also report the maximal leaves overlap with this route within the top-k predicted routes and consider a subset of 100 routes where all steps match templates seen during training.
Revised: Since this criterion is rather strong, we also report the maximal leaves overlap with this route within the top-k predicted routes and consider a subset of 100 routes where all steps match templates seen during training.

Original: Note that, interestingly, there is basically no difference between the top-5 and top-10 scores, so that we only show the former here.
Revised: Note that, interestingly, there is basically no difference between the top-5 and top-10 scores, so that we only show the former here.

Original: At first glance, we see that the performances are very different from the corresponding single-step experiments, and our selection of datasets reveals further differences.
Revised: At first glance, we see that the performances are very different from the corresponding single-step experiments, and our selection of datasets reveals further differences.

Original: Over the manually picked routes where all reactions match templates seen during training, the template-based models show much better performance than all other models.
Revised: Over the manually picked routes where all reactions match templates seen during training, the template-based models show much better performance than all other models.

Original: The results for the other datasets are extremely low reflecting the struggle of the models with distributions beyond the train data and more complex routes (esp.
Revised: The results for the other datasets are extremely low reflecting the struggle of the models with distributions beyond the train data and more complex routes (esp.

Original: rd).
Revised: rd).

Original: One main, additional problem we identified are the probabilities returned by the models, the search algorithms use those for cost estimates.
Revised: One main, additional problem we identified are the probabilities returned by the models, the search algorithms use those for cost estimates.

Original: In particular the template-free models based on common NLP frameworks return no normalized or regularized probabilities, assigning high ones to only to a few predictions, and hence prevent the search from exploring alternatives to their top suggestions altogether.
Revised: In particular the template-free models based on common NLP frameworks return no normalized or regularized probabilities, assigning high ones to only to a few predictions, and hence prevent the search from exploring alternatives to their top suggestions altogether.

Original: This is problematic given that their top-1 performance is not perfect.
Revised: This is problematic given that their top-1 performance is not perfect.

Original: We experimented with normalization and obtained best results (the ones shown) for all models by applying an additional softmax on the top-10 predictions given to AIZynthFinder.
Revised: We experimented with normalization and obtained best results (the ones shown) for all models by applying an additional softmax on the top-10 predictions given to AIZynthFinder.

Original: Note that (Tu & Coley, 2021) mention that additional engineering efforts are likely needed for G2S in the multi-step scenario, but the impact of such adaptations on both single and multi-step performance remains unclear.
Revised: Note that (Tu & Coley, 2021) mention that additional engineering efforts are likely needed for G2S in the multi-step scenario, but the impact of such adaptations on both single and multi-step performance remains unclear.

Original: The results for G2S further reveal a shortcoming of the top-k evaluation in PARoutes: the routes are sorted including route length as one criterion but, in this way, very unlikely predictions may be taken as top ones, especially with the template-free models.
Revised: The results for G2S further reveal a shortcoming of the top-k evaluation in PARoutes: the routes are sorted including route length as one criterion but, in this way, very unlikely predictions may be taken as top ones, especially with the template-free models.

Original: We experimented with atom mapping or a forward-direction model as suggested in (Schwaller et al., 2020) to validate the outcomes of the template-free models, and hence filter out unlikely predictions, but this causes considerable overhead.
Revised: We experimented with atom mapping or a forward-direction model as suggested in (Schwaller et al., 2020) to validate the outcomes of the template-free models, and hence filter out unlikely predictions, but this causes considerable overhead.

Original: In terms of metrics, we observe that the trends shown here are captured by most metrics with the exception of top-1 accuracy, and at different magnitudes.
Revised: In terms of metrics, we observe that the trends shown here are captured by most metrics with the exception of top-1 accuracy, and at different magnitudes.

Original: Our mss score is similar, but captures a different intent and some subtle difference (see previous subsection), in particular the large gap between the routes in rt and rd.
Revised: Our mss score is similar, but captures a different intent and some subtle difference (see previous subsection), in particular the large gap between the routes in rt and rd.

Original: Overall, NPP performs surprisingly well and is only outperformed by GLN; we could not find a specific reason for why it outperforms MHN but hypothesize that it is due to better regularization capability.
Revised: Overall, NPP performs surprisingly well and is only outperformed by GLN; we could not find a specific reason for why it outperforms MHN but hypothesize that it is due to better regularization capability.

Original: Lastly, in comparison to the experiments with the template-based models in (Genheden & Bjerrum, 2022), our results are much lower - as expected -, but also show much greater variation between the datasets.
Revised: Lastly, in comparison to the experiments with the template-based models in (Genheden & Bjerrum, 2022), our results are much lower - as expected -, but also show much greater variation between the datasets.

Original: This, again, seems to hint at a considerable impact of rare reactions, which occur more often in rd and turn out to be critical in its longer and more complex routes.
Revised: This, again, seems to hint at a considerable impact of rare reactions, which occur more often in rd and turn out to be critical in its longer and more complex routes.

Original: Discussion.
Revised: Discussion.

Original: • Generally, the identification of data aspects influencing predictions is critical for correctly interpreting the results .
Revised: • Generally, the identification of data aspects influencing predictions is critical for correctly interpreting the results .

Original: We show that the templates from the
Revised: We show that the templates from the

Original: train data reveal interesting insights
Revised: train data reveal interesting insights

Original: • The variety of models does not seem to lead to a large variety of predictions; in our experiments, the models predicted few truly unique solutions (i.e., usually less than 1% of the test set).
Revised: • The variety of models does not seem to lead to a large variety of predictions; in our experiments, the models predicted few truly unique solutions (i.e., usually less than 1% of the test set).

Original: • The main available search algorithms for the multi-step setting expect scaled and regularized probabilities , which are not given by all models.
Revised: • The main available search algorithms for the multi-step setting expect scaled and regularized probabilities , which are not given by all models.

Original: Hence it is not clear how those can be applied and how they perform.
Revised: Hence it is not clear how those can be applied and how they perform.

Original: Similarly, we need efficient techniques to estimate the validity of the predictions of template-free models.
Revised: Similarly, we need efficient techniques to estimate the validity of the predictions of template-free models.

Original: [Table]
Revised: [Table]

Original: • Currently, the template-based models are competitive with other models and, in the multi-step setting, the most simple and easy to use MLP still seems to provide one of the best solutions since it is both efficient and effective.
Revised: • Currently, the template-based models are competitive with other models and, in the multi-step setting, the most simple and easy to use MLP still seems to provide one of the best solutions since it is both efficient and effective.

Original: In our study, only GLN performs better.
Revised: In our study, only GLN performs better.

Original: • There is certainly major value in studying and comparing methods in a more theoretical context in the single-step scenario only, as it is usually done today.
Revised: • There is certainly major value in studying and comparing methods in a more theoretical context in the single-step scenario only, as it is usually done today.

Original: Our work supports such studies by showing the variation and commonalities in metrics beyond top-k accuracy.
Revised: Our work supports such studies by showing the variation and commonalities in metrics beyond top-k accuracy.

Original: Moreover, we propose the mss score, as a first attempt to capture other aspects relevant in route context.
Revised: Moreover, we propose the mss score, as a first attempt to capture other aspects relevant in route context.

Original: • The main open problem in terms of template-based models is the improvement in ranking .
Revised: • The main open problem in terms of template-based models is the improvement in ranking .

Original: While the latter is in focus of current research, this improvement should not happen at the expense of top-k for higher k’s given that multi-step prediction is the ultimate goal.
Revised: While the latter is in focus of current research, this improvement should not happen at the expense of top-k for higher k’s given that multi-step prediction is the ultimate goal.

Original: Here, MRR offers a good overall estimate, and the template-based models represent strong baselines.
Revised: Here, MRR offers a good overall estimate, and the template-based models represent strong baselines.

Original: • Generalization beyond reactions similar to the ones seen during training represents the other major challenge and seems to be solved only partly to date.
Revised: • Generalization beyond reactions similar to the ones seen during training represents the other major challenge and seems to be solved only partly to date.

Original: This leads to particularly low performance in the multi-step scenario, where a single new reaction leads to an overall failure.
Revised: This leads to particularly low performance in the multi-step scenario, where a single new reaction leads to an overall failure.

Original: Our work is intended to point out critical open issues rather than as a study of the SOTA.
Revised: Our work is intended to point out critical open issues rather than as a study of the SOTA.

Original: Considering recently proposed add ons (Fortunato et al., 2020; Sun et al., 2021; Lin et al., 2022), the actual
Revised: Considering recently proposed add ons (Fortunato et al., 2020; Sun et al., 2021; Lin et al., 2022), the actual

Original: SOTA is likely slightly better.
Revised: SOTA is likely slightly better.

Original: We cannot fully generalize from our results because we focused on representative models and data only, and the research prototypes we probed have not been optimized for the multi-step setting.
Revised: We cannot fully generalize from our results because we focused on representative models and data only, and the research prototypes we probed have not been optimized for the multi-step setting.

Original: Note that template-free models have been successfully integrated in tools with careful engineering (Schwaller et al., 2020), and advanced search algorithms can overcome shortcomings of single-step models; e.g., (Chen et al., 2020; Kim et al., 2021) obtain close to maximal performance with NeuralSym combined with an optimized cost estimate.
Revised: Note that template-free models have been successfully integrated in tools with careful engineering (Schwaller et al., 2020), and advanced search algorithms can overcome shortcomings of single-step models; e.g., (Chen et al., 2020; Kim et al., 2021) obtain close to maximal performance with NeuralSym combined with an optimized cost estimate.

Original:  Conclusions
Revised:  Conclusions

Original: We provide new datasets for retrosynthesis prediction which allow for evaluating prediction diversity, effectiveness in the multi-step context, and generalization, aspects that are usually not addressed when evaluating over USPTO-50k.
Revised: We provide new datasets for retrosynthesis prediction which allow for evaluating prediction diversity, effectiveness in the multi-step context, and generalization, aspects that are usually not addressed when evaluating over USPTO-50k.

Original: We further provide evaluation infrastructure.
Revised: We further provide evaluation infrastructure.

Original: Our evaluation of various single-step models suggests that the aspects our data addresses are critical for estimating multi-step performance.
Revised: Our evaluation of various single-step models suggests that the aspects our data addresses are critical for estimating multi-step performance.

Original: Altogether, our work shows that we need to evaluate more broadly to be able to address the challenges of retrosynthesis prediction in future research.
Revised: Altogether, our work shows that we need to evaluate more broadly to be able to address the challenges of retrosynthesis prediction in future research.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work started within a course project at UMass Amherst.
Revised: This work started within a course project at UMass Amherst.

Original: We thank Sahasra Iyer for initial discussions, and the tutors, especially Jay-Yoon Lee, and Andrew McCallum for all their support.
Revised: We thank Sahasra Iyer for initial discussions, and the tutors, especially Jay-Yoon Lee, and Andrew McCallum for all their support.

Original: A Model Configuration
Revised: A Model Configuration

Original: Single Step Models.
Revised: Single Step Models.

Original: For all models, we used the configurations suggested by the authors in the original works (see Section 2).
Revised: For all models, we used the configurations suggested by the authors in the original works (see Section 2).

Original: The details can also be found in our repository.
Revised: The details can also be found in our repository.

Original: NeuralSym(++).
Revised: NeuralSym(++).

Original: We ran hyperparameter tuning since the original paper (Segler & Waller, 2017) used different data; the final configuration(s) are detailed in the repository.
Revised: We ran hyperparameter tuning since the original paper (Segler & Waller, 2017) used different data; the final configuration(s) are detailed in the repository.

Original: We made some interesting observations in these experiments: (1) We also experimented with the more complex Highway network version described in (Segler & Waller, 2017), but the MLP provided better performance.
Revised: We made some interesting observations in these experiments: (1) We also experimented with the more complex Highway network version described in (Segler & Waller, 2017), but the MLP provided better performance.

Original: (2) We needed to add batch normalization to reach a performance comparable to the one reported in (Dai et al., 2019); this design choice was later confirmed in communication with chemists who have experience with template-based models and noted the importance of regularization, in particular, for the muti-step experiments.
Revised: (2) We needed to add batch normalization to reach a performance comparable to the one reported in (Dai et al., 2019); this design choice was later confirmed in communication with chemists who have experience with template-based models and noted the importance of regularization, in particular, for the muti-step experiments.

Original: (3) We also experimented with optimization, using the validation loss, top-1, top-5, and top-50 as criteria, respectively; interestingly, all these scenarios lead to basically the same hyperparameters (i.e., there was only some slight variation in the fingerprint dimension, 1028 vs. 2048, and layer number, 2 vs. 3).
Revised: (3) We also experimented with optimization, using the validation loss, top-1, top-5, and top-50 as criteria, respectively; interestingly, all these scenarios lead to basically the same hyperparameters (i.e., there was only some slight variation in the fingerprint dimension, 1028 vs. 2048, and layer number, 2 vs. 3).

Original: It is left open if this is due to the simplicity of the model or also holds for more complex ones.
Revised: It is left open if this is due to the simplicity of the model or also holds for more complex ones.

Original: Lastly, note that for the experiments over the larger rt and rd data, we disregarded templates for which there is only one instance in the training data.
Revised: Lastly, note that for the experiments over the larger rt and rd data, we disregarded templates for which there is only one instance in the training data.

Original: This was shown to be beneficial in (Seidl et al., 2022), and we observed this as well.
Revised: This was shown to be beneficial in (Seidl et al., 2022), and we observed this as well.

Original: Single Step Models - Influence of Beam Size.
Revised: Single Step Models - Influence of Beam Size.

Original: We observed some variation in performance with changed beam sizes.
Revised: We observed some variation in performance with changed beam sizes.

Original: However, since we could not identify a specific best value for the models - and even not for individual models - we mostly resorted to the settings suggested by the authors; we made an exception with G2S setting n_best=topk*3 since we experimented with larger topk than (Tu & Coley, 2021), which used a fix value.
Revised: However, since we could not identify a specific best value for the models - and even not for individual models - we mostly resorted to the settings suggested by the authors; we made an exception with G2S setting n_best=topk*3 since we experimented with larger topk than (Tu & Coley, 2021), which used a fix value.

Original: However, we found that since we also evaluated top-50, while some of the works considered maximally top-10.
Revised: However, we found that since we also evaluated top-50, while some of the works considered maximally top-10.

Original: We note that this also explains that there is no performance increase for GR in some cases from top-10 to top-50 (e.g., Table 5), but we obtained overall worse results for larger beam sizes and therefore used the ones originally proposed, keeping in mind that top-10 performance is likely more relevant in practice.
Revised: We note that this also explains that there is no performance increase for GR in some cases from top-10 to top-50 (e.g., Table 5), but we obtained overall worse results for larger beam sizes and therefore used the ones originally proposed, keeping in mind that top-10 performance is likely more relevant in practice.

Original: In particular, note that our multi-step experiments focus on maximally 10 predictions, hence this setting is in line with the ones from the original studies.
Revised: In particular, note that our multi-step experiments focus on maximally 10 predictions, hence this setting is in line with the ones from the original studies.

Original: Multi-Step Experiments.
Revised: Multi-Step Experiments.

Original: We used the default settings suggested in AIZynthFinder (see https://molecularai.github.io/aizynthfinder/configuration.html , accessed 08/23/2022) with the following three exceptions.
Revised: We used the default settings suggested in AIZynthFinder (see https://molecularai.github.io/aizynthfinder/configuration.html , accessed 08/23/2022) with the following three exceptions.

Original: We used Retro ∗ as search algorithm since we obtained extremely bad results with MCTS, probably because it uses the probabilities returned by the models in a way that does not fit their current, mostly unregularized scale.
Revised: We used Retro ∗ as search algorithm since we obtained extremely bad results with MCTS, probably because it uses the probabilities returned by the models in a way that does not fit their current, mostly unregularized scale.

Original: The suggested maximal branching (cutoff_number) of 50 resulted in infeasible runtimes with most non-template-based models, hence we chose 10.
Revised: The suggested maximal branching (cutoff_number) of 50 resulted in infeasible runtimes with most non-template-based models, hence we chose 10.

Original: We note that this was also due to our filtering of invalid molecule predictions.
Revised: We note that this was also due to our filtering of invalid molecule predictions.

Original: Since the numbers of the latter were extremely high with, for example, for CF, obtaining 50 valid reactant sets takes extremely long.
Revised: Since the numbers of the latter were extremely high with, for example, for CF, obtaining 50 valid reactant sets takes extremely long.

Original: Lastly, we increased the maximal number of reactions considered (max_transforms) to 10, which matches the maximal number in our data.
Revised: Lastly, we increased the maximal number of reactions considered (max_transforms) to 10, which matches the maximal number in our data.

Original: These adaptations show the need for considering practical aspects in model design and evaluation.
Revised: These adaptations show the need for considering practical aspects in model design and evaluation.

Original: B About USPTO-50k and USPTO-full
Revised: B About USPTO-50k and USPTO-full

Original: Datasets.
Revised: Datasets.

Original: The two most commonly used datasets are USPTO-50k (Schneider et al., 2016) and
Revised: The two most commonly used datasets are USPTO-50k (Schneider et al., 2016) and

Original: USPTO-full (Dai et al., 2019), both subsets of the US patent data (Lowe, 2017).
Revised: USPTO-full (Dai et al., 2019), both subsets of the US patent data (Lowe, 2017).

Original: As the names suggest, the latter, containing about 1M reactions, is much larger than the former; it is also used in much less works.
Revised: As the names suggest, the latter, containing about 1M reactions, is much larger than the former; it is also used in much less works.

Original: It has been observed recently, after our experimentation, that there are a few non-sensical data points in USPTO-50k (Lin et al., 2022); however, the split we used is still useful in that it eases comparison to approaches we did not include in our study since it is one of ones most commonly used.
Revised: It has been observed recently, after our experimentation, that there are a few non-sensical data points in USPTO-50k (Lin et al., 2022); however, the split we used is still useful in that it eases comparison to approaches we did not include in our study since it is one of ones most commonly used.

Original: We observed overlaps between the train and test data in USPTO-full and therefore did not experiment over this dataset.
Revised: We observed overlaps between the train and test data in USPTO-full and therefore did not experiment over this dataset.

Original: Results, Tables 4, 5.
Revised: Results, Tables 4, 5.

Original: The table gives an overview of what is known about the SOTA.
Revised: The table gives an overview of what is known about the SOTA.

Original: Our results over
Revised: Our results over

Original: USPTO-50k (over a common split) are similar, see Table 5.
Revised: USPTO-50k (over a common split) are similar, see Table 5.

Original: Our additional analysis, presented in this paper, completes the picture in terms of scalability and other aspects, such as generalization.
Revised: Our additional analysis, presented in this paper, completes the picture in terms of scalability and other aspects, such as generalization.

Original: Since not all models were originally evaluated over the same split (i.e., NPP, MHN, and CF(L)), we trained those.
Revised: Since not all models were originally evaluated over the same split (i.e., NPP, MHN, and CF(L)), we trained those.

Original: We re-evaluated all since this experiment served also to verify our wrapper implementations.
Revised: We re-evaluated all since this experiment served also to verify our wrapper implementations.

Original: MHN vs. NeuralSym, Table 6.
Revised: MHN vs. NeuralSym, Table 6.

Original: Not surprisingly, the more complex MHN outperforms NeuralSym.
Revised: Not surprisingly, the more complex MHN outperforms NeuralSym.

Original: However, the comparison between these models is not entirely fair since the settings are quite different.
Revised: However, the comparison between these models is not entirely fair since the settings are quite different.

Original: While NeuralSym uses rather standard features, MHN applies a 30k-dimensional vector of different fingerprints and additionally uses the validation data during training.
Revised: While NeuralSym uses rather standard features, MHN applies a 30k-dimensional vector of different fingerprints and additionally uses the validation data during training.

Original: In order to obtain a better estimate of the actual impact of MHN’s main feature, the template encoding, we therefore consider two other models, NeuralSym+ (NeuralSym + MHN’s initial features) and
Revised: In order to obtain a better estimate of the actual impact of MHN’s main feature, the template encoding, we therefore consider two other models, NeuralSym+ (NeuralSym + MHN’s initial features) and

Original: NeuralSym++ (NPP) (NeuralSym+ trained including the validation data).
Revised: NeuralSym++ (NPP) (NeuralSym+ trained including the validation data).

Original: And we see that the gap indeed shrinks.
Revised: And we see that the gap indeed shrinks.

Original: [Table]
Revised: [Table]

Original: C Details about USPTO-ms
Revised: C Details about USPTO-ms

Original: We used the raw data from (Dai et al., 2019) and basically followed their data cleaning steps.
Revised: We used the raw data from (Lowe, 2017) and basically followed their data cleaning steps.

Original: In a nutshell, we then collected all products for which there are different reactions - after dropping reagents - , and dropped those reactions that are contained in the train/valid data of USPTO-50k (w.r.t.
Revised: In a nutshell, we then collected all products for which there are different reactions - after dropping reagents - , and dropped those reactions that are contained in the train/valid data of USPTO-50k (w.r.t.

Original: the spit of (Coley et al., 2017; Dai et al., 2019)).
Revised: the spit of (Coley et al., 2017; Dai et al., 2019)).

Original: The exact steps are documented in a notebook in our Git repository.
Revised: The exact steps are documented in a notebook in our Git repository.

Original: D Details about rt & rd
Revised: D Details about rt & rd

Original: Datasets, Table 7, Figures 7, 8 We basically followed the procedure proposed by (Mo et al., 2021;
Revised: Datasets, Table 7, Figures 7, 8 We basically followed the procedure proposed by (Mo et al., 2021;

Original: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.
Revised: Genheden & Bjerrum, 2022) for extracting retrosynthesis routes from the USPTO data.

Original: The latter was obtained from (Thakkar et al., 2020), who provide a thoroughly cleaned dataset containing a reaction template with each reaction.
Revised: The latter was obtained from (Thakkar et al., 2020), who provide a thoroughly cleaned dataset containing a reaction template with each reaction.

Original: We note that, in this way, a filtering was done already (i.e., all reactions where the template extraction failed), yet we considered the template availability as valuable enough, especially for later analysis.
Revised: We note that, in this way, a filtering was done already (i.e., all reactions where the template extraction failed), yet we considered the template availability as valuable enough, especially for later analysis.

Original: However, we did not drop rare reactions (e.g., whose template appears less often in the data) and adapted the last step, in which we created two custom splits.
Revised: However, we did not drop rare reactions (e.g., whose template appears less often in the data) and adapted the last step, in which we created two custom splits.

Original: Our time split is based on the year data coming with the reactions.
Revised: Our time split is based on the year data coming with the reactions.

Original: For the split containing especially diverse routes as test and validation data we proceeded as suggested in (Genheden &
Revised: For the split containing especially diverse routes as test and validation data we proceeded as suggested in (Genheden &

Original: Bjerrum, 2022).
Revised: Bjerrum, 2022).

Original: E Details about n1 & n 
Revised: E Details about n1 & n

Original: Datasets, Table 8, Figures 7, 9.
Revised: Datasets, Table 8, Figures 7, 9.

Original: In order to see how general our data and results are, and to compare to the results from (Genheden & Bjerrum, 2022), we also experimented on their n1 and n5 datasets.
Revised: In order to see how general our data and results are, and to compare to the results from (Genheden & Bjerrum, 2022), we also experimented on their n1 and n5 datasets.

Original: Specifically, we extracted random test sets, of 1k and 5k routes for both n1 and n5, and evaluated our models trained over USPTO-50k over those.
Revised: Specifically, we extracted random test sets, of 1k and 5k routes for both n1 and n5, and evaluated our models trained over USPTO-50k over those.

Original: Note that we created these datasets in the same way as our rt and rd datasets, collecting all reactions contained in the considered routes.
Revised: Note that we created these datasets in the same way as our rt and rd datasets, collecting all reactions contained in the considered routes.

Original: Tableshows the numbers of reactions, and Figure 9 shows the shares of reactions that match templates from the USPTO-50k training data.
Revised: Tableshows the numbers of reactions, and Figure 9 shows the shares of reactions that match templates from the USPTO-50k training data.

Original: In this paper, we report the single-step results only for n and n5.
Revised: In this paper, we report the single-step results only for n and n5.

Original: We did not observe greater differences to the results over our datasets in the multi-step scenario and therefore did not complete the experiments, also due to the enormous resource requirements.
Revised: We did not observe greater differences to the results over our datasets in the multi-step scenario and therefore did not complete the experiments, also due to the enormous resource requirements.

Original: F Measuring Effectiveness: The Multi-Step Score (mss)
Revised: F Measuring Effectiveness: The Multi-Step Score (mss)

Original: Our multi-step score intends to complement existing metrics such as top-k accuracy by considering the route context.
Revised: Our multi-step score intends to complement existing metrics such as top-k accuracy by considering the route context.

Original: It is straightforward to calculate and interpret in that it captures the multi-step performance under the assumption that the model is combined with a perfect search algorithm.
Revised: It is straightforward to calculate and interpret in that it captures the multi-step performance under the assumption that the model is combined with a perfect search algorithm.

Original: Formally, let m k denote a single-step model which, given a target molecule, predicts a set of at most k sets of reactant molecules.
Revised: Formally, let m k denote a single-step model which, given a target molecule, predicts a set of at most k sets of reactant molecules.

Original: For a given tree structure representing a retrosynthetic route, with nodes representing reactions, let S denote the set of all those reactions and S
Revised: For a given tree structure representing a retrosynthetic route, with nodes representing reactions, let S denote the set of all those reactions and S

Original: i denote all reactions at level i (e
Revised: i denote all reactions at level i (e

Original: [Figure]
Revised: [Figure]

Original: g., S 0 represents the singleton set containing the first reaction step at the
Revised: g., S 0 represents the singleton set containing the first reaction step at the

Original: [Figure]
Revised: [Figure]

Original: #Molecules
Revised: #Molecules

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        100.
Revised: oo d        100.

Original: #Leaves train validtest rt-1k        100.
Revised: #Leaves train validtest rt-1k        100.

Original: #Reactions
Revised: #Reactions

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        100.
Revised: oo d        100.

Original: #Longestlinearroute      100.
Revised: #Longestlinearroute      100.

Original: #Molecules
Revised: #Molecules

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        100.
Revised: oo d        100.

Original: #Leaves train valid test rd-1k        100.
Revised: #Leaves train valid test rd-1k        100.

Original: [Figure]
Revised: [Figure]

Original: #Reactions
Revised: #Reactions

Original: L i k e l i h
Revised: L i k e l i h

Original: oo d        

Revised: oo d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: root node), such that S =
Revised: root node), such that S =

Original: (cid:83) i S i .
Revised: (cid:83) i S i .

Original: Then we compute mss as follows:
Revised: Then we compute mss as follows:

Original: [Equation]
Revised: [Equation]

Original: (P) (cid:59) , otherwise
Revised: (P) (cid:59) , otherwise

Original: T i + 1 : = T i ∪ (cid:40)(cid:83) R»P ∈ S i R,
Revised: T i + 1 : = T i ∪ (cid:40)(cid:83) R»P ∈ S i R,

Original: if ∀ R»P ∈ S i : R ∈ m k (P) and T i − 1 ̸= T i (cid:59) , otherwise mss k ( m ,S 0 ,S 1 ,...) :
Revised: if ∀ R»P ∈ S i : R ∈ m k (P) and T i − 1 ̸= T i (cid:59) , otherwise mss k ( m ,S 0 ,S 1 ,...) :

Original: = | T
Revised: = | T

Original: [Table]
Revised: [Table]

Original: | | S |
Revised: | | S |

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Note that in our evaluation, we consider a form of micro average, considering the predictions for all routes in the test set at once, and compute roughly (cid:80) T | T ∞ | (cid:80) S | S | .
Revised: Note that in our evaluation, we consider a form of micro average, considering the predictions for all routes in the test set at once, and compute roughly (cid:80) T | T ∞ | (cid:80) S | S | .

Original: For ease of computation, we also regard all nodes at one level at once and do not differentiate branches.
Revised: For ease of computation, we also regard all nodes at one level at once and do not differentiate branches.

Original: This processing can be considered in line with an interactive setting, which explores the tree breadth-first, one level at a time.
Revised: This processing can be considered in line with an interactive setting, which explores the tree breadth-first, one level at a time.

Original: G Additional Results
Revised: G Additional Results

Original: Results, Tables 9, 10, 11, 12, Figure 10.
Revised: Results, Tables 9, 10, 11, 12, Figure 10.

Original: We extracted random subsets of n1 and n5 of two sizes, 1k and 5k routes, to see how the results over the 1k sets generalize.
Revised: We extracted random subsets of n1 and n5 of two sizes, 1k and 5k routes, to see how the results over the 1k sets generalize.

Original: The tables show that there are no considerable differences for all models.
Revised: The tables show that there are no considerable differences for all models.

Original: Figure 10 shows that all models struggle on the new reactions; in particular, there are no great differences between top-1 and top-10, the latter are only slightly larger.
Revised: Figure 10 shows that all models struggle on the new reactions; in particular, there are no great differences between top-1 and top-10, the latter are only slightly larger.

Original: Tables 11 and 12 are discussed in the main paper.
Revised: Tables 11 and 12 are discussed in the main paper.

Original: For GR, we obtained very disappointing numbers.
Revised: For GR, we obtained very disappointing numbers.

Original: For verification purposes, we therefore also ran the GR version trained over USPTO-50k and this returned much better results; a possible reason might be that the vocabularies constructed based on our larger training sets are too large and confuse the model (while the vocabulary size for USPTO-50k is 174, we have 289 for rt and 307 for rd), so it might be possible to obtain better results by dropping less common structures.
Revised: For verification purposes, we therefore also ran the GR version trained over USPTO-50k and this returned much better results; a possible reason might be that the vocabularies constructed based on our larger training sets are too large and confuse the model (while the vocabulary size for USPTO-50k is 174, we have 289 for rt and 307 for rd), so it might be possible to obtain better results by dropping less common structures.

Original: Also note that we set a train time limit of 24 hours, as it was done in (Irwin et al., 2022).
Revised: Also note that we set a train time limit of 24 hours, as it was done in (Irwin et al., 2022).

Original: In particular NPP and CF might benefit from increasing that.
Revised: In particular NPP and CF might benefit from increasing that.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: H Further Discussion
Revised: H Further Discussion

Original: • We show that the templates from the train data reveal interesting insights and also observed a slight correlation between performance on reactions matching training templates and the number of template instances in the training data for all models (15-25%
Revised: • We show that the templates from the train data reveal interesting insights and also observed a slight correlation between performance on reactions matching training templates and the number of template instances in the training data for all models (15-25%

Original: Pearson correlation, depending on the model).
Revised: Pearson correlation, depending on the model).

Original: We also checked for a correlation between molecule complexity and prediction performance but did not observe any in our experiments.
Revised: We also checked for a correlation between molecule complexity and prediction performance but did not observe any in our experiments.

Original: Reaction complexity might be a factor worth to study in future work.
Revised: Reaction complexity might be a factor worth to study in future work.

Original: • One problem with the non-template-based models is that they often produce invalid molecular structures or reactions.
Revised: • One problem with the non-template-based models is that they often produce invalid molecular structures or reactions.

Original: While the former is technically no problem because those can be easily identified with tools such as RDKit, such a filtering step may yield large runtime overhead.
Revised: While the former is technically no problem because those can be easily identified with tools such as RDKit, such a filtering step may yield large runtime overhead.

Original: Moreover, there is few efficient technology for identifying invalid reactions.
Revised: Moreover, there is few efficient technology for identifying invalid reactions.

Original: In order to get an idea of the validity of the predicted reactions, we briefly experimented with
Revised: In order to get an idea of the validity of the predicted reactions, we briefly experimented with

Original: forward reaction prediction models as suggested in (Schwaller et al., 2020) and atom mapping tools for post processing in the multi-step experiments, but this was extremely time consuming as well.
Revised: forward reaction prediction models as suggested in (Schwaller et al., 2020) and atom mapping tools for post processing in the multi-step experiments, but this was extremely time consuming as well.

Original: Given this trade-off, it is likely that non-templatebased models will only be competitive if they either optimize for validity (amongst others) or include efficient post processing steps, to provide more efficient models; or if they show explicit benefits over template-based models, justifying the increase in runtime.
Revised: Given this trade-off, it is likely that non-templatebased models will only be competitive if they either optimize for validity (amongst others) or include efficient post processing steps, to provide more efficient models; or if they show explicit benefits over template-based models, justifying the increase in runtime.

Original: • While the trends in the paper tend to show that the template-based models are most effective to date, we emphasize that they are limited by nature and will never allow for generalization.
Revised: • While the trends in the paper tend to show that the template-based models are most effective to date, we emphasize that they are limited by nature and will never allow for generalization.

Original: Although the current models do not seem to solve this task satisfactory yet, we see some generalization capability.
Revised: Although the current models do not seem to solve this task satisfactory yet, we see some generalization capability.

Original: It will be interesting to see, how much generalization is possible with state-of-the-art ML technology once this aspect is considered explicitly in future studies.
Revised: It will be interesting to see, how much generalization is possible with state-of-the-art ML technology once this aspect is considered explicitly in future studies.

Original: We also note that there has been related work focusing on the generalization capability of the transformer across chemical space showing promising results (Lee et al., 2019); our study complements this type of investigation in pointing out reaction
Revised: We also note that there has been related work focusing on the generalization capability of the transformer across chemical space showing promising results (Lee et al., 2019); our study complements this type of investigation in pointing out reaction

Original: templates as efficient means to quantify generalization, even for non-template-based models.
Revised: templates as efficient means to quantify generalization, even for non-template-based models.

Original: I Limitations
Revised: I Limitations

Original: Our study focuses on more general aspects of evaluating retrosynthesis prediction and, in the course of this, provides an exemplary evaluation.
Revised: Our study focuses on more general aspects of evaluating retrosynthesis prediction and, in the course of this, provides an exemplary evaluation.

Original: As it is in the nature of experimental studies, there are various other possible and interesting settings which can and sometimes definitely should be explored in future work.
Revised: As it is in the nature of experimental studies, there are various other possible and interesting settings which can and sometimes definitely should be explored in future work.

Original: Below, we point out aspects which should be kept in mind when interpreting our results.
Revised: Below, we point out aspects which should be kept in mind when interpreting our results.

Original: • It has been noted in the past that the USPTO data is lacking certain kinds of reactions and is biased towards certain kinds of reaction (Schneider et al., 2016), hence it may not provide sufficient samples to train ML models useful for practice.
Revised: • It has been noted in the past that the USPTO data is lacking certain kinds of reactions and is biased towards certain kinds of reaction (Schneider et al., 2016), hence it may not provide sufficient samples to train ML models useful for practice.

Original: While this is likely the case, our study shows that the state of the art still struggles on the available data, in particular, in terms of generalization.
Revised: While this is likely the case, our study shows that the state of the art still struggles on the available data, in particular, in terms of generalization.

Original: Hence, the USPTO data seems to provide enough challenges for ML for the current moment.
Revised: Hence, the USPTO data seems to provide enough challenges for ML for the current moment.

Original: • We considered several template-based models but only two different template-free ones, and GraphRetro, while there are many others for each model class.
Revised: • We considered several template-based models but only two different template-free ones, and GraphRetro, while there are many others for each model class.

Original: It will be interesting to see if our observations regarding these representatives are similar for other models in future studies.
Revised: It will be interesting to see if our observations regarding these representatives are similar for other models in future studies.

Original: • For most experiments, we used the models trained over USPTO-50k and the (hyper)parameter settings suggested in the original studies, since benchmarking the SOTA was out of the scope of this study.
Revised: • For most experiments, we used the models trained over USPTO-50k and the (hyper)parameter settings suggested in the original studies, since benchmarking the SOTA was out of the scope of this study.

Original: We experimented with some parameters, such as beam sizes, and obtained slight variations in the results; however, for the parameters we considered, these variations did not lead to considerable changes in the overall trends.
Revised: We experimented with some parameters, such as beam sizes, and obtained slight variations in the results; however, for the parameters we considered, these variations did not lead to considerable changes in the overall trends.

Original: • In the multi-step experiments we fixed the settings using mostly the default settings from AIZynthFinder and Retro ∗ as search algorithm.
Revised: • In the multi-step experiments we fixed the settings using mostly the default settings from AIZynthFinder and Retro ∗ as search algorithm.

Original: We also resorted to creating the stocks representing the buyable molecules as suggested in (Genheden & Bjerrum, 2022).
Revised: We also resorted to creating the stocks representing the buyable molecules as suggested in (Genheden & Bjerrum, 2022).

Original: Certainly, variation in all these settings would lead to changes in the results.
Revised: Certainly, variation in all these settings would lead to changes in the results.

Original: Nevertheless, our evaluation often shows some considerable and more general differences between the models, which will likely generalize to some extent.
Revised: Nevertheless, our evaluation often shows some considerable and more general differences between the models, which will likely generalize to some extent.