Paper: Re-evaluating Retrosynthesis Algorithms with Syntheseus (ICLR-2024)

Reviewer comment (Experiments):
Some important baseline methods are not included in Figure 1, such as [1, 2] which show strong performance and also provide source code at https://github.com/uta-smile/RetroComposer and https://github.com/Jamson-Zhong/Graph2Edits, respectively.

Author response:
**(A1)**: Thank you for the suggestion; we recently learned about Graph2Edits and were considering integrating it into `syntheseus`. We have now confirmed that the model works under our default single-step environment, and wrote a wrapper aligning it with the shared model interface. We will post precise evaluation results soon and then add them to Figure 1 of our paper.

We finished testing Graph2Edits through our `syntheseus` wrapper on USPTO-50K test set. The results in comparison to the originally reported numbers are as follows:

|| top-1 | top-3 | top-5 | top-10 | top-50 |
|---|---|---|---|---|---|
| original paper | 55.1% | 77.3% | 83.4% | 89.4% | 92.7% |
| with `syntheseus` | 54.6% | 76.6% | 82.8% | 88.7% | 91.7% |

We found that the results in the re-evaluation are roughly similar but slightly worse (by around 0.6%) than those reported in the original paper. We will include these results in our manuscript but first take a closer look to see if there is a clear explanation for the small differences in performance.

Indeed, we are also looking at RetroComposer. Integrating Graph2Edits was quicker, as the open-source repository already provides a trained checkpoint.

Reviewer comment (Evaluation):
Suggest careful considerations on the suggested post-processing procedures. Is it true the best practice should be to only consider valid molecules when computing top-k accuracy? I agree with the authors that invalid molecules can be removed easily. However, the ability to generate valid molecules is also important. Otherwise, the model might need to generate multiple invalid molecules before a valid molecule, which consumes more time compared to the case where the first generated molecule is valid. Same as the duplicated outputs.

Author response:
**(A3)**: Indeed, measuring the fraction of valid/unique results is useful, and (as we mention above), this can be computed using the model wrappers present in `syntheseus`. For end-to-end evaluation, we recommend discarding invalid molecules and duplicates; if a model is asked for e.g. 50 results, but due to this filtering it is only left with 10, then this will be reflected in it’s top-k accuracy for `k > 10`, we do not re-query the model to provide more outputs until 50 are gathered. However, if desired, it is simple to implement this re-querying procedure using our `ReactionModel` abstraction. `syntheseus` does record the total time taken by model calls (something previous works usually did not account for), making it is easy to quantify the effect of such adaptive inference on model speed.

Reviewer comment (Reproducibility):
The only question would be the quality and ease of the Python package SYNTHESEUS. While some suggested practices are indeed important and help to remove unfair evaluation, I would suggest keeping some practices optional and configurable, such as removing invalid or duplicate outputs.

Author response:
**(A2)**: Deduplication can be turned off by setting `skip_repeats=False` in the evaluation script. We don’t have an option to turn off the removal of invalid molecules, as the results are returned as `Molecule` objects which are guaranteed to be valid (this makes a lot of the code more robust/safe as it doesn’t have to check for invalid SMILES). However, it is possible to assess the fraction of valid predictions by comparing how many results were returned vs how many were requested. So both the fraction of valid results as well as the amount of duplicates can be computed using `syntheseus`.

Reviewer comment (Evaluation):
The SYNTHE-SEUS library falls short in addressing the core evaluation challenges within retrosynthesis. It continues to rely on top-k accuracy without incorporating other significant metrics such as precision and recall, which may offer a more comprehensive evaluation about algorithm performance.

Author response:
**(A2):** We discuss in the manuscript that top-k accuracy is equivalent to recall. We also discuss precision as a metric, and suggest that round-trip accuracy using a forward model can be used as a surrogate for it, which `syntheseus` has support for. However, as we say in the paper, precision is not a well-defined metric without a feasibility model, and all such models are imperfect, so the best you can really hope for is to provide a _framework_ that can compute precision with a given model. In `syntheseus` the forward model is abstracted away using a similar interface as the backward model, so the framework is flexible in terms of which model is used.

Reviewer comment (Reproducibility):
Although the authors purport that SYNTHE-SEUS is intended to serve as a resource for researchers developing retrosynthetic methods, there is a lack of clarity in Section 3 on how the platform will facilitate the flexible incorporation of different feature definitions, such as functional groups and molecular fingerprints.

Author response:
**(A3):** In our library, the interface for a single-step retrosynthesis model is (roughly) a function that accepts a molecule as a SMILES string, and returns a list of suggested reactions to make this molecule, representing the reactants in each reaction again as SMILES strings. This means any kind of model that can parse SMILES strings can be used. For example, `rdkit` could be used to convert the SMILES into a molecule object, from which one can compute fingerprints or extract functional groups.

Reviewer comment (Evaluation):
Can the authors discuss the multi-step planning method FusionRetro [1] and its proposed evaluation metric? If it makes sense, can the authors evaluate it and include its metric in the experiment or library?

Author response:
**(A1)**: Thank you for the suggestion. We carefully reviewed [1] and found the approach proposed there very interesting and relevant. Thus, we will cite this publication in the discussion section of our paper. We have also reviewed the “set-wise exact match” metric, and drafted utility functions that make computing it easy. We expect to merge the relevant PR into `syntheseus` soon (pending some testing and code review). We will also add support for search algorithms like FusionRetro into our roadmap for the library (however, properly integrating it will take much longer than the length of the rebuttal period).

Reviewer comment (Reproducibility):
Can the authors provide an overview of the library, e.g., supported single-step models, search algorithms, evaluation metrics etc?

Author response:
**(A2)**: The `syntheseus` library (snapshot of which can be accessed through the supplementary material) is, at the top level, split into the following subdirectories:
- `reaction_prediction`, which defines inference wrappers for all supported single-step model types (7 models mentioned and evaluated in the paper, plus an 8th model currently being integrated as requested by Reviewer JYEf) and a evaluation script `eval.py`, which computes metrics such as: top-k accuracy, MRR, time spent on model inference (excluding e.g. post-processing or model loading, leading to a more precise measurement), round-trip accuracy using a forward model (proxy for precision) and number of model parameters.
- `search`, which defines all supported multi-step search algorithms (MCTS for both OR and AND-OR graphs, Retro*, and PDVN). The search algorithms are implemented in such a way that the commonalities between them are extracted into common base classes, making it easy to build further algorithms that e.g. reuse the logic for expanding the search graph present in MCTS but modify the node selection criteria.
- `interface`, which defines data structures at the intersection of single-step and multi-step, making sure they use the same objects for compatibility; notably, `interface/models.py` contains the shared `ReactionModel` interface.
- `cli`, which defines a top-level search entry point for running end-to-end retrosynthesis.

There are already several `*.md` files in the repository explaining this structure, and a `tutorials` directory with notebooks showcasing how `syntheseus` can be used. On top of this, we are also working on hosting documentation in the form of a static webpage to make it even easier to set up and use the library.

Reviewer comment (Experiments):
The methods included in the re-evaluation are not enough as a benchmarking library. Some common baselines like vanilla LSTM, and vanilla Transformer should be included. Also, more state-of-the-art methods are welcome.

Author response:
Incorporating baselines in a proper way is significant engineering work, as we do not only run them, but standardize the (conda) environments and dependencies, and wrap using our interface to integrate fully into `syntheseus`. Thus, we are careful to select the baselines to maximize usefulness. The older models are written using legacy frameworks and old tensorflow and pytorch versions, while with RootAligned and Chemformer we have 2 Transformer baselines already, and these papers showed that these models are superior to “vanilla” Transformer models or even LSTMs. Thus, we believe the effort integrating more baselines would be best spent on newer models beating state-of-the-art, rather than old baselines which were already outperformed by many newer approaches. We encourage the community to integrate such models via PRs in our repository if desired.

Reviewer comment (Experiments):
What are the criteria for selecting state-of-the-art methods?

Author response:
We selected the initial set of 7 methods based on a combination of state-of-the-art performance (this is why we leaned towards more advanced Transformer-based models rather than their vanilla counterparts) and coverage of varying model types (e.g. end-to-end, template-based, sequence-of-edits-based). That being said, `syntheseus` is not a “final product” but a library that we will continue to develop (no matter the ICLR review outcome) and open it up to the community, so more methods will be added in the future.

Reviewer comment (Novelty):
What are the advantages of Pistachio over USPTO-FULL? It would be better if this issue is discussed in the main paper.

Author response:
There are several advantages of Pistachio:
- significantly larger than USPTO-FULL and with higher coverage of diverse reaction
- of high-quality (commercial and carefully curated), and much less noisy
- continuously updating every few months

In comparison, USPTO is now a bit outdated (1976-2016), and suffers from noise and quality issues caused by automated toolkits (e.g. Indigo) and outdated software that were used to prepare it (see “Recent advances in artificial intelligence for retrosynthesis” for more discussion). Including it would require significant additional preprocessing to give robust results, which would make comparisons to any works using prior versions of USPTO-FULL invalid. That being said, looking at USPTO-FULL can still be worthwhile, and some researchers may not have access to Pistachio. Due to that, we decided it is useful for us to report results on Pistachio as the comparison may be valuable and it is something most researchers cannot compute themselves. Results on USPTO-FULL and other datasets can be easily generated as `syntheseus` now supports evaluation data in several formats, including the format commonly used for USPTO.