Paper: RetroDiff: Retrosynthesis as Multi-stage Distribution Interpolation (ICLR-2024)

Reviewer comment (Evaluation):
The diffusion models usually require more time. In this paper, the two stage diffusion models could be less efficient than other baselines in terms of the inference time.

Author response:
Admittedly, diffusion models have a common drawback compared to other end-to-end models. However, the inefficiency of the two-stage diffusion model inference is not as pronounced as it may seem, thanks to task specificity. For external bond generation, we require only a few steps to complete the inference (as few as 50 steps in the paper's setting), owing to the small distributional difference between the two sides.

Reviewer comment (Evaluation):
For the top-1 accuracy, the performance gain of the proposed method is marginal. However, it outperforms in top-3, 5, and 10 accuracy.

Author response:
Although there is only a marginal improvement in top-1 accuracy, it is essential to recognize that the significance of top-3/5/10 accuracy is on par with that of top-1. Consequently, the overall competitiveness of our method remains robust.

Reviewer comment (Reproducibility):
How to preprocess the dataset to get the intermediates right before the post-adaption?

Author response:
For external groups: The dataset employs a numbered mapping of atoms that remain unchanged before and after the reaction. 
Consequently, external groups can be automatically distinguished by identifying unnumbered atoms. For external bonds: After removing the external group from the product, a direct comparison between the product and the reactant allows us to identify the broken bond. All these operations can be carried out using RDKit.

Reviewer comment (Experiments):
Can the proposed post-adaption cover all the reactions?

Author response:
It can cover approximately 95% of reactions (we have verified that the percentage of special cases is within reasonable bounds).

Reviewer comment (Novelty):
The method is incremental: the diffusion models are taken off-the-shelf, and also the overall approach of group/bond generation is already seemingly known. It’s difficult to see why this method works well, and I suspect it’s mostly transformer tuning. This is not elaborated in the paper. Finally, the method is poorly motivated and novelties are vague. I think this is a successful engineering effort towards solving an important problem, but scientifically there is very little contribution.

Author response:
Thank you for highlighting the motivational and creative nuances in our approach. We have refined the introductory section to enhance the clarity and robustness of our motivation. In essence, our motivation unfolds in two key dimensions:

1. Enhancing Semi-Template Methods: Semi-template methods offer scalability[1,2] (compared to template-based approaches), interpretability and diversity[3] (compared to template-free methods). However, current semi-template methods fall short in performance. Our goal is to pioneer a suite of more efficient semi-template methods, capitalizing on their inherent advantages.

2. Innovating with Diffusion Models for Retrosynthesis: Conceptualizing the retrosynthesis as a distribution interpolation challenge, we identify diffusion models as a promising architecture for probabilistic modeling. However, existing templates overly constrain the intrinsic data structure, necessitating artificial modifications to molecular structures—a hindrance to feasible distribution transformations. To overcome this, we have redefined the templates, tailoring them to the modeling requisites of distribution transformations.

Addressing this complex problem goes beyond a successful engineering effort, as employing diffusion models directly proves impractical. Transformer tuning alone lacks intrinsic effectiveness. 

Our scientific contribution lies in redefining the chemical template, aligning it with the modeling requirements of the task, thereby maximizing the capabilities of the diffusion model.

Reviewer comment (Writing):
The introduction is vaguely written and I couldn’t follow what are the open problems this work tackles, or what are its contributions or their motivations. Many sentences are just impossible to understand. A good example of this is the incomprehensible sentence: “However, directly following the chemical reaction tem-plate could render the property of exploration on the distribution transformation of the diffusion model also be contrast to the data inherent structure by adding artifacted order to the motif and bonds.” (??) The intro also claims that autoregressive models are bad (yet this paper proposes a diffusion model), and that accumulating errors is somehow good. I don’t understand what the paper means by “reset”.

Author response:
Thank you for highlighting the crucial aspects concerning our contributions and motivations. We have meticulously revised the introductory section to amplify both clarity and robustness in articulating our motivation. In response to W2, we have provided detailed elaborations.

To address incomprehensible sentences, including the one you flagged, we have taken proactive measures to rectify them. Furthermore, we acknowledge and rectify the inappropriate inclusion of the cumulative error of the autoregressive model as a motivation, opting for its removal.

Additionally, the term "reset" has been accurately adjusted to "redefine" to signify our deliberate redefinition of the chemical prior. This adjustment underscores our commitment to tailoring the model to the distributional transitions, making a distinct contribution to the field of semi-template methods.

Reviewer comment (Theory):
The f is unlikely to be invertible, so the notation f^{-1} seems quite adhoc.

Author response:
Thanks for pointing out our laxity in mathematical formalization. We have removed the ambiguous mappings f and f^{-1} to make the task definition clearer (see Section 2 for modified details).

Reviewer comment (Theory):
The f(p()) notation seems nonsensical. You are mapping a density value to something. This does not seem to make sense. Similarly, the f( p*p/P_X ) is also nonsense: you can’t divide a density evaluation with a distribution, and then map it to something.

Author response:
Thanks for pointing out our laxity in mathematical formalization. We have rewritten notations of the task definition and the template setup (see Section 2 for modified details), you can check it and feel free to ask new questions if any.

Reviewer comment (Presentation):
I don’t understand fig2. So we start with 3 balls in the left: what are these? Then they become 3 colored balls with one edge. What does this mean? Are the balls atoms and colors the atom types?

Author response:
Thanks for pointing out the lack of clarity in our presentation. We have modified the interpretation of Figure 2 to make the information in the figure more understandable. Specifically, balls are atoms and colors are atom types. The 3 white balls on the left mean 3 dummy atoms, and the colored balls mean the 3 dummy atoms have been denoised into real atom categories. About the elaboration of dummy atoms, you can see section 2.2 (have updated) for more details.

Reviewer comment (Experiments):
How do you know how many extra atoms to add (the external group)? What does “noisy” external group mean?

Author response:
Thanks for pointing out the lack of clarity in our presentation. The number of extra atoms is set as a constant, and we set a dummy atom category for indirectly judging the actual number (for example, the unexisted atom is set as the dummy category, you can see updated section 2.2 for details). The"noisy" external group are all dummy atoms with no bonds.

Reviewer comment (Theory):
Eq 6 feels nonsensical: what does it mean to have a loss between x and p(x)? One is a graph, another is a scalar.

Author response:
Thanks for pointing out the presentation ambiguity. Eq 6 (now Eq 8) was intended to calculate the cross-entropy of the ground truth with the logic output by the network, but there is ambiguity in formalization. We have modified that loss function to avoid ambiguity, as shown in Eq 8.

Reviewer comment (Experiments):
Strong baselines are missing: for example RetroKNN [1] on the template-based side, and RootAligned SMILES [2] on the template-free side. Both of these models outperform RetroDiff by a large margin (a few %) across top-k values. Note that recent work [3] corrected some of the previously reported results on USPTO-50K, showing that e.g. results for LocalRetro and RetroKNN were somewhat inflated, but even after the correction RetroKNN is stronger than RetroDiff.

Author response:
Thanks a lot for mentioning the related works. It should be noted that the scope of our work focuses on the semi-template methods where tradeoff between the scalability and interpretability is emphasized. As the reviewer suggested, we have added the mentioned baselines and the discussion on their difference in the model class with our model in the updated version for the completeness of the empirical comparison. 

As you noticed, in terms of performance, our method falls short of the template-based RetroKNN. However, our approach attains the SOTA performances in the semi-template methods and holds a noteworthy advantage over the template-based methods. For more details, please refer to the response to (c).

Reviewer comment (Experiments):
The authors focus their comparison within semi-template methods, noting how their performance is strong within that class. This seems a bit artificial, as belonging to the semi-template class doesn't convey strong benefits in itself that would warrant not looking at other model classes. One could argue that template-free methods are too unconstrained and not interpretable, and hence not focus on them in a potential comparison, but template-based methods are usually at least as interpretable as semi-template-based ones, and sometimes more interpretable. In consequence, performance among all reaction models overall shows RetroDiff doesn't perform too favourably.

Author response:
- Firstly, we acknowledge that the performance of Retrodiff is not on par with the current SOTA template-based method, Retroknn. However, the difference is not insignificant (retroknn's top-1 result is 55.3%, 2.7% higher than ours). Furthermore, it is crucial to reiterate that we lead as the SOTA among semi-template methods.

- Nevertheless, the most significant drawback of the template-based approach, as we have previously highlighted in the introduction, is its inherent scalability disadvantage. This limitation has been extensively discussed in prior research, and we substantiate this point by citing arguments from other papers:

  - RetroXpert[1]: "The reported synthetic organic knowledge consists of in the order of 10^7 reactions and compounds[5]... It is infeasible to manually encode all the synthesis routes in practice considering the exponential growth in the number of reactions[6]... An obvious limitation is that these methods can only infer reactions within the chemical space covered by the template database, preventing them from discovering novel reactions[7]."

  - G2G[2]: "Despite their great potential for synthesis planning, template-based methods, however, not only require expensive computation but also suffer from poor generalization on new target structures and reaction types."

  - Retroformer[3]: "Despite the state-of-the-art accuracy and guaranteed molecule validity, these methods are limited to the scope of the existing template database."

  - MEGAN[4]: "Due to computational limitations, they typically require representing reactions using a restricted number of templates, which necessarily limits the coverage of the chemical reaction space accessible by such methods."

- In general, stepping back to survey the landscape of prior research, the template-based approach indeed boasts performance advantages, but its notable scalability issue remains a persistent challenge. Conversely, the semi-template approach stands out with two key strengths: scalability and interpretability, making a distinct and valuable contribution. These two approaches can run in parallel rather than as opposing forces, each bringing significant value to the research community. We hope our contributions are duly acknowledged. Thank you once again!

Reviewer comment (Evaluation):
The authors show improved SMILES validity score compared to popular models based on the Transformer architecture. This is a good sanity check, although this result should not be given too much weight, as many highly performant template-based models (e.g. LocalRetro, RetroKNN) have 100% validity by design. It is still good to see RetroDiff getting near-perfect validity, but this limitation is actually absent in some model classes.

Author response:
Thank you for bringing this to my attention! I acknowledge that the template-based approach excels in achieving legitimacy due to the comprehensive templates. Nevertheless, our RetroDiff approach is also highly proficient in this aspect. While not a point for celebration, it underscores the robustness of our model.

Reviewer comment (Experiments):
In Section 2.1.3 on post-adaptation, the authors explain that in the final stage, zero or one bond is broken in the transformed product graph to form the reactant graphs. This appears to assume there are at most two reactants. While this may be true in USPTO-50K, it is not true overall. In larger reaction databases examples with 3 or even 4 reactants exist. If the current post-adaptation procedure only works if there are 2 reactants, this limitation should be highlighted as it would be a significant hindrance to real-world usability.

Author response:
**Regarding the generalizability of post-adaptation to more complex retrosynthesis datasets:**

The adaptability of post-adaptation is scalable. To illustrate, when dealing with 3 reactants, there are 4 reaction sites. Therefore, it becomes crucial to determine whether these 4 sites form two sets of adjacent sites to establish their legitimacy. This principle holds true for scenarios involving more than 3 reactions.

Reviewer comment (Writing):
Several parts of the work are not clear, preventing full understanding of some of the modelling or algorithmic details – see the "Questions" section for concrete examples. Finally, the writing needs improvement to make the paper reasonably easy to read (see the "Nitpicks" section for concrete suggestions).

Author response:
Thank you for your detailed writing suggestions. We have carefully addressed the points in your nitpick list, making revisions to enhance the overall quality of our writing. Regarding your three questions, we provide the following explanations:

> Q1: Is Equation 4 fully mathematically correct? I am a bit confused by the notation and the disappearance of the integral.

Thank you for meticulously scrutinizing the mathematical details of the paper! We have rewritten the "Denoising" part in section 2.1.1. The mathematical derivation is now more detailed and accurate.

> Q2: Could you elaborate on "We have obtained a trained network $p_{\theta}$ in the last stage, so we freeze g and x in the graph and continue to train $p_{\theta}$."? Are the overall networks trained in stages utilizing different training "datasets"?

Thanks for pointing out the lack of clarity in our presentation. To be precise, different targets were employed, as elaborated below:

In the first stage, the input comprises a graph with a noisy external group and the true product, where no bond exists between the two (i.e., a dummy category). The output is the true external group.

In the second stage, the input is also a graph, but now it consists of the true external group and the true product, with noisy external bonds connecting the two. The output is the true external bond. 

It's important to note that the graph structure remains identical in the two stages, with the only difference being whether different parts of the input graph are noisy or true.

> Q3: Could you elaborate on Section 2.2? I feel like the reader needs to fill in a lot of details to understand it, e.g. infer the exact structure of a1, b1 and b2 (how many ones/zeros); it is also confusing that these are used to define v1 and v2 before being properly introduced themselves.

Thanks for pointing out the lack of clarity in our presentation. We have rewritten section 2.2 to make it easy to understand. Ambiguous notations, such as a1, b1, and b2, have been replaced, and detailed explanations have been added to enhance clarity.

Reviewer comment (Experiments):
several baselines were missing

Author response:
Thanks for pointing out the lack of a baseline. We apologize for neglecting this recent work, and we will discuss it further in the related work.

Reviewer comment (Experiments):
why are so many complex steps and rule checking needed? a good generative model would learn all of that from the data. have the author ablated this properly?

Author response:
We didn't design very complicated models. Instead, we defined a new chemical template to implement the simplest distributional transformations on this basis.

The previous templates imposed numerous constraints, demanding human intervention in atom and bond manipulation during reactions, which causes the incompatibility of generative models. We redefined the templates, breaking down the task into two serial stages: external group generation and external bond generation. This two-stage approach simplifies the problem and aligns it with the application scenario of diffusion modeling, allowing for the direct learning of distributions.

And the last step of RULE CHECKING is just a simple check based on the legitimacy of the chemical compounds, and is not a complex predefined rule.