Paper: Retro-fallback: retrosynthetic planning in an uncertain world (ICLR-2024)

Reviewer comment (Experiments):
More quantitative experimental results and baseline comparisons need to be clarified.

Author response:
Thanks for reading our paper and providing helpful comments. We will respond to your comments and questions below. Please also refer to our general response ([link](https://openreview.net/forum?id=dl0u4ODCuW&noteId=GbZ5C1tSlR)).

**Weakness 1: more results**

We have performed _over 6000_ new search comparisons on the retro* and FusionRetro test datasets to strengthen our quantitative results. The results are shown in sections F.4-F.5 and confirm that retro-fallback is more effective at maximizing SSP than other algorithms while being no worse with respect to other metrics.

We believe that these additional results are more than enough to support our claims. We remind the reviewer that previous papers in multi-step retrosynthesis (e.g. retro*, DFPN-E, PDVN) were accepted to similar conferences despite performing fewer experiments.

The baselines for comparison are described in detail in sections F.1.4-F.1.5. Essentially we did our best to configure some of the most common baseline algorithms for multi-step retrosynthesis to optimize SSP (knowing it is not possible to directly set SSP to be the "objective"). We are happy to answer any specific questions you may have about them.

Reviewer comment (Evaluation):
This paper has a solid motivation and a novel solution to perform the route planning but is limited by the proposed SSP metric. SSP attempts to calibrate the feasibility of chemical reactions. However, it will not perform better than a forward one-step prediction model since they are trained on the similar reaction dataset. Besides, SAScore is a trivial metric as it prefers unrealistic large carbon rings.

Author response:
Thanks for reading our paper and providing helpful comments. We will respond to your comments and questions below. Please also refer to our general response ([link](https://openreview.net/forum?id=dl0u4ODCuW&noteId=GbZ5C1tSlR)).

**Weakness 2: limits of SSP metric**

We think the reviewer may have misunderstood our SSP metric. First, SSP is not a single specific metric, but rather a way to translate the predictions of a reaction feasibility model into a metric for entire synthesis plans. Its values depend on an underlying reaction feasibility model, and therefore it is not an *alternative* to a forward one-step prediction model, but would likely be *implemented* using such a model.

Second, SSP does not depend on SAScore in any of our experiments. We used SAScore as a _search heuristic_ to guide our algorithms, but never to evaluate the quality of the synthesis plans found during search. We agree it is trivial and has unrealistic preferences which make it unsuitable to evaluate route quality.

Reviewer comment (Experiments):
The quantitative results seem not adequate enough to support the findings, could the author present more quantitative results?

Author response:
Thanks for reading our paper and providing helpful comments. We will respond to your comments and questions below. Please also refer to our general response ([link](https://openreview.net/forum?id=dl0u4ODCuW&noteId=GbZ5C1tSlR)).

**Weakness 1: more results**

We have performed _over 6000_ new search comparisons on the retro* and FusionRetro test datasets to strengthen our quantitative results. The results are shown in sections F.4-F.5 and confirm that retro-fallback is more effective at maximizing SSP than other algorithms while being no worse with respect to other metrics.

We believe that these additional results are more than enough to support our claims. We remind the reviewer that previous papers in multi-step retrosynthesis (e.g. retro*, DFPN-E, PDVN) were accepted to similar conferences despite performing fewer experiments.

The baselines for comparison are described in detail in sections F.1.4-F.1.5. Essentially we did our best to configure some of the most common baseline algorithms for multi-step retrosynthesis to optimize SSP (knowing it is not possible to directly set SSP to be the "objective"). We are happy to answer any specific questions you may have about them.

Reviewer comment (Novelty):
While I appreciate the general formulation and systematic discussion about planning algorithms, given the existence of retro*, the novelty and advantage of this work (e.g., performing parallel updates using multiple samples) is not that obvious to me. I would encourage adding more discussion when talking about detailed method.

Author response:
**Weakness 1: novelty/advantage of this work over retro\***

There are two ways to view the advantage and novelty of retro-fallback over retro\*:

1. Objective: retro-fallback optimizes the SSP objective (i.e. probability that at least one synthesis route is successful) while retro* minimizes a scalar cost. Critically, _there is no way to set the costs in retro* so that it optimizes SSP_. An example which illustrates this well is when there are correlations between reaction outcomes (e.g. either reaction A or reaction B will work, but not both). This cannot be represented by assigning individual scalar costs to reactions A and B. Therefore retro-fallback _can optimize objectives which retro* is incapable of._
2. Mechanism: retro-fallback performs $k$ independent updates and aggregates them to estimate a conditional expectation, while retro* uses just a single cost function. Retro-fallback is a more complicated algorithm than retro\*.

We have revised the comparison of retro* vs retro-fallback in section 5 to hopefully make this distinction more clear.

Reviewer comment (Novelty):
Since this work focuses on multi-step retrosynthesis, related works [1] working on this should be discussed.

Author response:
**Weakness 2: FusionRetro not discussed**

Thank you for pointing us to this work, we were not aware of it. FusionRetro was a very interesting paper that proposes a hybrid single-step model and multi-step search algorithm, as well as a new metric. We have added a citation to this work and dedicated a section in the appendix to discussing it (G.4). We have also added an experiment using their benchmark dataset and metric in apprix F.5 (described below and in our general response).

Reviewer comment (Reproducibility):
How does Algorithm 1 generate multiple possible reaction graph for a target molecule?

Author response:
**Question 1: how does algorithm 1 generate multiple reaction graphs**

Much like retro*, proof number search, or MCTS, retro-fallback does not directly generate synthesis plans: instead, it grows an explicit search graph which contains the synthesis plans implicitly. A second enumeration step is required afterwards to extract the synthesis plans. Algorithm 1 just states the procedure for growing the search graph but omits this second step. We omitted it because other works (e.g. retro*) also do not discuss it and because enumerating paths in a graph can be done with relatively simple depth-first or breadth-first search algorithms.

Reviewer comment (Evaluation):
Algorithm 1 could generate plan that is not a valid plan, e.g., when the algorithm terminates, there are still tip nodes in G'. To take such cases into consideration, can the authors also evaluate performance using two other metrics: the success rate as in Retro*, and the set-wise exact match accuracy as in FusionRetro?

Author response:
**Question 2: invalid plans/extra experiments**

Our answer to Q1 should hopefully clarify this. Retro-fallback will almost certainly leave tip nodes in G, but the secondary algorithm to enumerate synthesis routes will skip over any routes whose end nodes are not purchasable. Therefore it is not possible to output invalid plans.

Nonetheless we still performed an experiment with the FusionRetro dataset/metric as the reviewer suggested. Essentially we ran retro-fallback and our baseline algorithms on the FusionRetro test set using one of our existing feasibility models. The results are shown in appendix F.5 and suggest that retro-fallback achieve a higher SSP score while performing similarly on the FusionRetro exact matching metric.

We further note that the “success rate” metric was already included in the paper under the name “fraction solved” (see figures F.6-F.7). Retro-fallback consistently outperformed the baseline algorithms according to this metric and did so on the FusionRetro benchmark too.

Reviewer comment (Experiments):
It is interesting to see retro-fallback with transformer can be more computational efficient, and the authors elaborate more on that (e.g., how to combine both for improvement)?

Author response:
**Question 4: combining with transformer**

Good question: we think the best potential to combine our method with a transformer is to use it as the single-step reaction prediction model or feasibility model. By not relying on templates, such a model could be very flexible. We discuss combining retro-fallback and FusionRetro specifically in appendix G.4. This is an exciting direction for future research!

Reviewer comment (Novelty):
The reviewer could not find the key difference between retro-fallback and the two existing algorithms, DFPN-E [Kishimoto+ 2019] or Retro* [Chen+ 2020]. If negative log probabilities are used as the proof or disproof number, retro-feedback, Retro*, and DFPN-E seem very similar. The reviewer may not have understood the intended meaning of the sentence: "The key difference is that retro-fallback performs parallel updates...". What does "parallel" mean in this case? Could the authors elaborate on this part?

Author response:
**Q1 (key difference between retro-fallback and existing algorithms DFPN-E and Retro\*)**

There are two ways to view the differences. The first is in the objective: retro-fallback aims to find a _set_ of synthesis plan which act as backup plans for each other (thereby maximizing our SSP metric) while retro*/DFPN-E aim to find a _single_ plan which minimizes some scalar cost. Importantly, this cost _cannot_ simply be set to SSP (we explain this in more detail in section 4.1 and appendix C). As an intuitive example, imagine that either reaction A or reaction B will be feasible, but not both: this dependency cannot be captured by a scalar cost. _Therefore, retro-fallback is pursuing a fundamentally different objective than the existing algorithms mentioned in your review._

The second way to view the differences is mechanistically. The main difference is that retro*/DFPN-E all perform a _single_ cost update in an AND/OR graph while retro-fallback performs _$k$ separate updates_ (one for each sample from the stochastic processes). This is what we meant by “parallel updates”. Although the individual update equations are equivalent to those of retro* in one special case, they are not equivalent in general. As far as we can tell, the updates are not equivalent to DFPN-E in any cases.

In summary, although these algorithms all perform heuristic-guided search on an AND/OR graph, their objectives and mechanisms are distinct, so we think the algorithms are quite different. We revised some text in section 5 to clarify this (e.g. we removed the confusing term “parallel updates”). Please let us know if this addresses your concerns.

Reviewer comment (Reproducibility):
The reviewer might have misunderstood something, but the problem described in Section 4.1 could be easily avoided if the authors used a hash table for implementing the search algorithms. Using a hash table is a standard technique for proof-number search. If a hash table is used, a cycle could be easily detected, making it possible to avoid problems caused by the same molecule or reaction appearing multiple times in a path.

Author response:
**Q2 (hash table)**

We believe the reviewer may be asking about section 4.2 rather than section 4.1 because cycles are not discussed in section 4.1.

If considering just a single synthesis plan, we agree that hash tables can be useful to check whether a cycle is present and potentially to remove it (in fact we use hash tables in our analysis code to enumerate valid synthesis plans). However, the problem described in section 4.2 is to calculate the recursively-defined quantities $\rho$ and $\psi$ for all nodes in the graph. We _do not_ calculate this by enumerating all plans (that would scale poorly because a graph can contain a combinatorially large number of paths). Instead we advocate for a dynamic-programming style update which will run in polynomial time. Our comment at the end of the section merely states that because the graph contains cycles, we cannot do a simple dynamic programming update in linear time. The graph contains cycles because we do not include duplicate molecule/reaction nodes like some previous works. At no point in the updating process are individual synthesis plans considered or enumerated. Therefore we don’t see how a hash table would resolve this issue.

Please let us know whether this answers your question, and in particular whether you really did mean to write section 4.1.

Reviewer comment (Novelty):
The statement on page 1 — "Although existing algorithms may find multiple synthesis plans, they are generally not designed to do so, and there is no reason to expect the plans found will be suitable as backup plans (e.g. they may share steps with the primary plan and thereby fail alongside it)" — appears to overlook related work. The following paper proposes an AND-OR-tree-based search algorithm for retrosynthesis, which keeps enumerating (probably) all the synthetic routes one by one in the order of some preference:  
Shibukawa, R., Ishida, S., Yoshizoe, K. et al. CompRet: a comprehensive recommendation framework for chemical synthesis planning with algorithmic enumeration. J Cheminform 12, 52 (2020). https://doi.org/10.1186/s13321-020-00452-5

Author response:
**Q3 (multiple plans)**

The CompRet paper you referenced considers the problem of _extracting_ individual synthesis plans from a search graph, while retro-fallback (and retro*, MCTS, DFPN-E) are all algorithms to _construct_ a search graph. Algorithms like CompRet are required to extract the routes afterwards. We used a simple best-first search algorithm to do this but will look into CompRet as an alternative (and have added a citation to this paper). Thanks for the reference!

Reviewer comment (Novelty):
There is an older paper that proposes to use AND-OR tree search for retrosynthesis. Please consider referring to this paper:  
Heifets, A., & Jurisica, I. (2021). Construction of New Medicines via Game Proof Search. Proceedings of the AAAI Conference on Artificial Intelligence, 26(1), 1564-1570. https://doi.org/10.1609/aaai.v26i1.8331  
https://ojs.aaai.org/index.php/AAAI/article/view/8331

Author response:
**Minor comments**

- Older paper: thanks for the reference, we have added a citation in section 2. We were not aware of this paper.

Reviewer comment (Theory):
Is it reasonable to assume there is always only one product?

Author response:
**Minor comments**

- Assuming one product: all existing multi-step retrosynthesis algorithms (e.g. retro*, DFPN-E) make this assumption, even if they do not state it explicitly. We think it is reasonable since many reactions have a single main product, and multi-product reactions can be represented as two separate reactions (e.g. A+B -> C+D can become A+B->C and A+B->D).