From: ICLR.cc/2023/Conference/Paper4082/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper4082/Reviewer_iHCh
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper describes an RL-based approach to generate novel reactants for optimizing the reaction yields. The proposed method uses a predictor to estimate the reaction yields, whose prediction is used as the reward in RL. Due to the limited amount of reaction data, the predictor is trained following a transfer learning approach. The source task is to pretrain a language model to predict the next character in a SMILES string and the target task is to predict the reaction yield. 

The method incorporates an RNN-based generator in its RL framework to produce new SMILES string. The generator is only used to design a partial molecule, a fragment, and the generated fragments are fused with a fixed core structure (specifically selected for each reaction) to form the final molecules. The parameters of RNNs are updated in the RL framework to achieve biased molecule generation aiming for higher reaction yield. 

The model is evaluated on three reaction types, and achieves state-of-the-art performances comparing to existing baselines.
strength_and_weaknesses: - Strength
1. The paper achieves a state-of-the-art performance in reactant generation with higher reaction yield.
2. The proposed model uses transfer learning for reaction yield prediction, which alleviate the difficulties caused by small reaction datasets.
3. It also propose to add a pre-selected core structure to the generated molecules to maintain the reacting site. 

- Weaknesses
1. This paper adopts an RL-based approach to generate molecules with higher yields. This kind of strategy has been used in molecule design fields which aims to optimize different properties. (e.g. REINVENT uses policy-based RL to tune RNNs for generating SMILES with optimized bioactivities; ReLeaSE also uses RL to generating molecules with higher reaction yields with a random forest based yield predictor.) It seems the major difference in this paper coming from using a transfer learning-based reaction yield predictor, which may not be novel enough from a technical perspective.
2. The author uses a simple approach to combine the generated fragment and the core, which choose the first H containing atom in the generated molecule and replace the H with the core fragment. However, this operation cannot guarantee that the fused molecule is synthesizable. 
3. The authors only compared the proposed methods with its own unbiased version for reaction b and reaction c. It would be more fair to also compare with other baselines as in the case of reaction a.
clarity,_quality,_novelty_and_reproducibility: One major concern about this work is its technical novelty as discussed in the weakness section. 
In addition, there are several unclear parts in this paper. 

1. In results and discussion, the authors compared TL-UB with Rxn-UB and claims “Although the *VUN* values for Rxn-UB model are lower than the TL-UB baseline model, the increase in $\overline{y}$ from 56.5 to 63.8, is an important aspect”. If I understand correctly, Rxn-UB is a generic RNN-based SMILES generator, which is not optimized for any reaction yield. The only reaction-specific operation is adding a reaction specific core to the generated SMILES. Therefore, whether adding the core structure is the main contributor of the higher yield of Rxn-UB? Can the authors provide more justification for these results?
2. Is the yield reported in table 1 calculated using the TL-based predictor? Does the ReLeaSE-B model use the same predictor in its biased generation setting?
3. In the supplemental info, the author only reports the RMSEs of the TL-based predictors. It would be more convincing if the author can compare its performance with the predictors used in ReLeaSE and TL-UB. 
4. What strategies are used in the baselines to maintain the core reaction sites? 
5. What is the major differences between ReLeaSE and Rxn since they are both RL-based approach?
 

summary_of_the_review: In summary, this paper proposes an RL-based approach for reaction yield optimization and achieves state-of-the-art performance on three reaction types. The proposed RL-based optimization scheme has been explored in other molecular generation fields and the model lacks technical novelty from this perspective. In addition, the paper still needs more performance comparison and analysis, such as comparing ReLeaSE and TL-UB performance for reaction b and c and also justifying the contribution of the core structure in reaction yield.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4082/Reviewer_rUiL
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: - De novo molecular design (also sometimes referred to as inverse design) is applied to find optimal reactants for reactions.
- A SMILES-based RL algorithm is applied for this purpose

strength_and_weaknesses: Stengths
- novel application of de novo molecular design methods (RL on SMILES is probably the most widely used method in practice)
- simple model, which seems to work well enough

Weaknesses
- grandiose language, and introduction with is disconnected from the impact of the rest of the paper, like typical organic chemistry papers ("multi-pronged", seriously?)
- doesn't use the latest and greatest of ML modelling (i.e. equivariant GNNs, or operating directly on graphs), but maybe this is a good thing
- innovation somewhat limited from the ML side, but an interesting application nevertheless
clarity,_quality,_novelty_and_reproducibility: 1) the paper is mostly clearly written
2) the quality is appropriate for a decent scientific paper
3) presentation is clear
4) while de novo design is not new, applying it to reagent discovery is understudied, and interesting; new aspects are included to ensure diversity of the generated structures
summary_of_the_review: - interesting, original application
- limited novelty on the method side
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4082/Reviewer_vM46
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed a REINFORCE based framework for optimizing molecules that lead to high yield in synthetic chemistry. 
strength_and_weaknesses: Strengths:
1. The authors explored DL and RL based molecular optimization methods specifically for reaction discovery. The discovery would be interesting to synthetic chemists. Clearly, it is an application paper. 

Weaknesses: 
1. The methodology, i.e. the DL based framework, lacks novelty in terms of algorithm development. The base machine learning models used in this framework are all well studied and used. 
2. The Fragmentation & Joining component of the framework involves a core-fragments. How do these core-fragments formed? Are they task specific?
3. It is unclear how the reward function is defined, as the regressor seems to have multiple outputs. 
4. When comparing molecular optimization methods with molecular generation methods, it is not surprising to see that molecular optimization methods yields better performance in terms of the desired properties. In the recent years, there were lots molecular optimization methods based on DL, RL, and other computational intelligence methods. However, the comparison in this paper is only restricted to ReLeaSE. 
5. Furthermore, the presentation of this paper can be improved by clarifying some important details, such as the meaning of the regressor output, and the data sets .

clarity,_quality,_novelty_and_reproducibility: The manuscript needs clarifications in some important details. The proposed model itself lacks novelty. However, it could be interesting for synthetic chemists.
summary_of_the_review: Overall, a tailored DL framework is developed in this paper in inverse molecular design. This work can be viewed as an application paper. Contributions to methodology are low. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4082/Reviewer_xGhN
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper proposes using reinforcement learning for reaction discovery. The task of reaction discovery is a lot less developed than more common tasks in ML for molecules as molecular property prediction or molecular design. The approach utilizes REINFORCE to generate SMILES of substituents on chemicals (reactants, catalysts) that maximize the yield of a reaction, quantified with an externally trained regressor, trained through fine-tuning of a pre-trained LLM on randomly selected molecules. They test the approach on three reactions (each of which has ~10^3 training points), seeking to maximize the predicted reaction yield 
strength_and_weaknesses: Strengths 
The approach seems straightforward and well-described on the basis of existing tools. 
The task of "reaction discovery", i.e. yield-optimization is relatively new and might be an interesting addition to the ML for chemsitry tool box and draw interest from other contributors. 

Weaknesses. 
The use of REINFORCE or other RL techniques for optimization SMILES-based representations of molecules for a given task is very well established. Although the authors call their approach reaction discovery, the task is a molecular optimization task, they are looking for a molecule that optimizes a certain property. The property being the yield in the reaction of the newly-designed molecule with fixed, pre-chosen chemicals. So in that sense this is indistinguishable from previous molecular design papers, only the prediction task has a larger input than the molecule being designed, and also needs the other chemicals, but these are kept fixed. Without moving in all the chemicals at the same time, there is no difference from existing approaches

It is well known that generative models in general, and RL based ones in particular, since it is so expressive, are too powerful for most property prediction models and end up essentially "poking holes" in the support of the predictor function. Specially in these case, where the predictors have seen a few dozen molecules at train time ,there's no reason to expect that they generalize well, so the predictors are in all likelihood completely overfit and easily defeated by the RL approach that just produces molecules different enough from the predictor trainin data that that "break" it and achieve high prediction but could be completely nonsensical. Some uncertainty quantification, or optimizing with a model trained of on data fold and testing with a model trained on another would perhaps quantify this issue. As of now, it is impossible to distinguish this work from one in which the predictors are completely overfit and the optimizer is just feeding them random molecules that won't behave at all as predicted. 

The task of reaction discovery is not well defined, and reactions 2 and 3 are "catalyst discovery" which is a common thing - trying to get a given desired chemical in the best possible way. Reaction 1 is confusing, because changing the substrate would also change the product, meaning that to get the reaction to "work well" one is ready to just make some random molecule. This task is confusing, since the molecules that make the reaction run well may be completely irrelevant for one's application. 

Something seems off in the double bond pattern of the leftmost blue molecule in Figure 5d. There is a 4-connected aromatic nitrogen and a two-connected one. 

Why is reaction 3 not in the Figure 5
clarity,_quality,_novelty_and_reproducibility: The paper reads clearly and the steps taken are well communicated. The task is somewhat novel but that take on it is too close to prior literature on molecular design
summary_of_the_review: This work takes the RL-based design of molecule SMILES with a twist by generating reactants or catalysts that maximize reaction yield. The task and method are analogous to abundant literature and there little evidence that the model is producing meaningful predictions since the predictor piece of the optimization loop is likely overfit.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4082/Reviewer_Vswi
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper is interested in the problem of reaction discovery. In particular, for a given reaction (with a fixed set of conditions and all but one of the reagents defined) this paper designs a method (called Rxn-B) for designing the final reactant to maximize the overall yield. Additionally, the method ensures the final reactant contains a defined core fragment (e.g., a hydroxyl group). 

Rxn-B works by initially training both (1) a generative model of SMILES strings based on RNNs  (Section 2.2 and pp.12-13) as well as (2) a predictor of yield ("surrogate model") based on a pre-trained language model (Section 2.1 and p.13). The REINFORCE algorithm is then used to finetune the weights of the generative model of SMILES strings (using the predictive model trained in the first part to define the reward) such that it generates molecules with better yield (again judged by the predictive model trained in the first part).


## Contributions
The contributions of the paper are as follows:
1. The paper describes the problem of reaction discovery and highlights how the use of ML methods has been relatively underexplored in the task of substrate scope expansion (compared to their popularity in modeling chemical/biological properties of molecules).
2. The paper adapts and updates popular techniques for optimizing molecules for biological/chemical properties (Olivecrona et al., 2017) so that they can be used for optimizing a reactant given a core fragment.
3. The paper shows how this method can find reactants that lead to better yield (as judged by a predictive model) compared to randomly sampling from an unbiased generative model of molecules (trained on molecules from a subset of the CHEMBL database).

strength_and_weaknesses: ### S1:  Interesting problem &  interesting future directions
I think the paper tackles an interesting (and relatively underexplored) problem and it's nice to see work that focuses on generative/optimization algorithms for reaction discovery rather than just property optimization. There are also interesting future directions (such as simultaneously or sequentially optimizing other reactants), which the authors discuss briefly in Section 6.

### S2: Approach produces high scoring molecules
Putting aside for now the concerns I have about whether the optimization target is very useful (see W1 below), it does seem the proposed approach is able to find molecules that do well against the target, while also being diverse and valid.


### W1: Evaluation may just exploit the surrogate model, limiting the approach's practical value
My understanding is that both the RL algorithm's reward function and the final evaluation of proposed reactants' yields depend on the same surrogate model. I'm worried in such a scenario the optimization algorithm can just exploit the surrogate model, finding "adversarial" molecules where the predicted scores can have little bearing on reality (see Renz et al., 2019). 

Moreover, this concern is compounded by the fact that the surrogate model only gets trained on ~100-1000 reactions to begin with (e.g., for reaction A only 740 reactions are used and of these only 37 have different alcohols -- the molecule that the model is trying to change). Therefore,  I fear that the surrogate model might not even be that accurate to begin with. Ideally, these fears could be addressed with some real-world evaluation, but failing that, some analysis of whether the model could rediscover held out experimental data would be nice.

Renz, P., Van Rompaey, D., Wegner, J.K., Hochreiter, S. and Klambauer, G. (2019) ‘On failure modes in molecule generation and optimization’, Drug Discovery Today: Technologies, 32-33, pp. 55–63.

### W2: Technical approach seems very similar to previous work
The technical approach of pre-training a generative model of SMILES strings on a large database of molecules and then finetuning it using an RL algorithm to optimize for a particular task has been well explored, e.g.: Segler et al., 2017, Olivecrona et al., 2017, Popova et al., 2018, Brown et al., 2019. 
The main difference to these works seems the use of a language model for the surrogate model instead of a random forest/actual oracle, which does not seem a large change. Therefore, I think the current paper has limited technical novelty.


### W3:  Weak baselines.
Only two baselines are considered (TL and ReLeaSE) and even these are only compared against for reaction A. Why are these approaches not tried for Reactions B and C? And what about some of the other approaches cited in Section 2?

clarity,_quality,_novelty_and_reproducibility: # Clarity
I thought the paper was clear and did a good job explaining the proposed approach (Figures 1-3 were helpful in this regard). In general, the results were well presented but I think the different reactions could be made more distinct in Table 1 (Figure 4a is much better in indicating the differences through the use of color).

# Quality
Unfortunately, I have concerns about the quality of the experiments (see W1 and W3 above). Therefore, I think the overall quality of this work currently has room for improvement.

# Novelty
I think the novelty of the proposed algorithmic approach is low, as it bears strong similarity to previous approaches (see W2 above). Applying  such algorithms to reaction discovery provides some task novelty but I find it hard to judge the significance of this due to issues in the experimental evaluation described above (W1 and W3).

# Reproducibility
Overall, I think the authors did a good job with respect to reproducibility: Jupyter notebooks are provided to run the experiments and the appendices describe some of the key experimental details.

# Clarifying Questions
**Q1.** Are you surprised with the difference between ReLeasSE-UB and Rxn-UB in Table 1? I thought (from Section 4.2) that these models used very similar generators and so in the unbiased setting are expected to perform very similarly? Is the only difference the enforcement of the core fragment in the Rxn-B approach?  
**Q2.** Is there any intuition for the validity of the molecules increasing in Table 1 when you switch from Rxn-UB to Rxn-B?   
**Q3.** What are the best yields reported in the original training dataset for Table 1?  
**Q4.** I assume specifying a core fragment is to ensure that the reactant still reacts with the other molecules in the same way? Have you thought about removing this constraint?  
**Q5.** At the top of p.14 you report the RMSEs for the yield predictor. Thanks for these! What are the training/test split sizes and how consistent are these values over different folds? I also have a hard time visualizing what a number such as 7.28 means (this seems quite high...?), have you calculated $R^2$ values?  

summary_of_the_review: Although I think the paper tackles a very interesting problem (that of reaction discovery and in particular substrate scope expansion), I do not think it is currently ready for publication and have therefore gone with a lower overall score. The main issue is that I am concerned that by evaluating the optimized molecules using the same surrogate model used in the optimization algorithm, one cannot be confident that the suggested molecules work well in practice (weakness W1 above). Furthermore, aside from using a different optimization function,  the method is very similar to previous work (W2).

## Rationale behind the other scores below:
- **correctness:** the method is evaluated empirically and as I mentioned above there is a real risk that the method simply returns molecules that exploit the surrogate model so I do not believe the claims are currently well supported. 
- **technical novelty/significance:** the method bears similarities to work that optimizes molecules for chemical/biological properties (see W2 above).
- **empirical novelty/significance:** molecular optimization algorithms are often not used for reaction discovery and therefore I think this paper has some empirical novelty. However, the concerns I have about correctness means that I find it hard to currently judge the empirical significance. Moreover, only one of the three reaction classes considered has the results of the baselines reported (see W3 above). I would like to see these issues addressed before increasing my significance score.

correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough