Paper: A SIMILARITY-AGNOSTIC REINFORCEMENT LEARNING APPROACH FOR LEAD OPTIMIZATION (ICLR-2024)

Reviewer comment (Evaluation):
Lead optimization in drug discovery is an important and difficult task. I have difficulty accepting the method as an invention or improvement for lead optimization. For my understanding lead optimization is a much more complicated process than purely looking on QED score or a similarity score, which the authors didn’t investigate.

Author response:
We refer the reviewer to Common Response (1).

As mentioned, our method is a generalized approach for generating alternate candidates to the lead molecule. Therefore, we have not explored the subsequent step of docking or binding affinity in the context of our work.

Reviewer comment (Experiments):
The paper would gain strength if the authors would compare their method against more recent methods in generative design and more properties other than QED. Especially, the baseline seems to be quite weak with all the efforts recently put into improving generative methods.

Author response:
We refer the reviewer to Common Responses (2) and (3). Results for the added baseline and metrics can be found in the Results Table in the Common Response.

Reviewer comment (Novelty):
The second contribution of their paper as stated on page 2 says: “we propose a search strategy…”. Could the authors elaborate more on the search strategy? In case it is just generating thousands of molecules and sorting them based on a score, this does not seem like a novel strategy.

Author response:
The reviewer is referring to the following statement in the paper:

“We propose a search strategy that separates the property optimization from the training and offloads it as a post-curation process, thereby simplifying the task of learning.”


We acknowledge the comment made by the reviewer. The novelty comes from the idea of using the search strategy to offload the property optimization from the training procedure due to the problems associated with multi-objective optimization.

To explain further, we take the example of the reward function used in MolDQN [1] for multi-objective optimization:

$r_t = w \times SIM(m_t, m_0) + (1-\omega) \times QED(m_t)$

The optimal value of ‘w’ in this multi-objective case is hard to determine quantitatively during hyperparameter tuning. In fact, the notion of optimality may be task-dependent and may very likely not be well understood for most tasks. If the user is not satisfied with the generated molecules, a new model would need to be trained with a different ‘w’. As opposed to this, in our work, the model would not need to be re-trained. Only the much cheaper search strategy needs to be employed to generate a large number of molecules that can be filtered to retain desired properties. Other multi-objective optimization challenges, such as Pareto dominance, conflicting objectives, etc are also avoided in our method.

Reviewer comment (Writing):
The last paragraph of section 4.6., “we found that under the condition G(a_t leads to g)…”, was difficult to understand. Could the authors elaborate more on the issue they observed?

Author response:
Thank you for the question. We had noticed that based on different values of the negative rewards that we chose, the policy would either converge or diverge.

  

To recap,

-   In our method, we select a fixed number L of negative return trajectories per high return trajectory.
    
-   The policy gradient loss moves the function away from low-return areas and closer to high-return areas.
    

Therefore, if the magnitude of the negative reward were high, the cumulative magnitude of gradient from the negative samples in a batch would overwhelm the magnitude of positive samples, causing it to diverge. The condition we mentioned works as a normalizing factor for the negative component of the loss and was essential for reward design in our case to ensure convergence.

Reviewer comment (Experiments):
Did I understand it correctly that the offline dataset just contains molecules randomly put together using the reaction rules, so potentially not chemically plausible at all?

Author response:
We are unsure if we understood the question correctly. The trajectories in the offline dataset are indeed put together using a random policy **given the reaction rules**. This still makes them chemically feasible though.

We elaborate for further clarification: Say we are given a start molecule. To generate a trajectory for the offline dataset, first, a reaction rule will be selected randomly among those that are applicable to it. This will result in a product which is chemically feasible. even though it is the result of a random sampling. By repeating this step multiple times, we generate a trajectory for our dataset using a random policy where each molecule is chemically feasible.

Reviewer comment (Theory):
My understanding of actor-critic reinforcement learning is to use the output of the critic for the loss of the actor. From eq. (1) and (2) this seems not the be the case, could the authors elaborate a little bit?

Author response:
The reviewer has noticed correctly. Due to our unique setup, we were able to use calculated returns to update the actor, thus not requiring the higher variance estimation by the critic.

  

The critic is still a crucial part of our method. Firstly, it provides additional gradients to update the embedding module, resulting in higher-quality representations. Secondly, it is used in the generation procedure to sort the actions suggested by the policy.

Reviewer comment (Evaluation):
The significance of the work is not clear. The method is trained to optimize molecules towards the target structures, but I am unsure if I understand how this model could be used in practice. Usually, the goal of lead optimization is to improve a set of molecular properties without impacting binding affinity. In the presented setup, the optimization changes the structure of lead candidates to make them more similar to known drugs, which oftentimes is undesired because only novel structures can remain outside the patented chemical space.

Author response:
> "The significance of the work is not clear.  The method is trained to optimize molecules towards the target structures, but I am unsure if I understand how this model could be used in practice. Usually, the goal of lead optimization is to improve a set of molecular properties without impacting binding affinity."

We refer the reviewer to Common Response (1).  
Having not considered a target receptor, we did not discuss binding affinity. We thank the reviewer for pointing it out. We acknowledge that it would be an important step after our method for further curation.

Reviewer comment (Experiments):
The experimental section seems very preliminary. Only two RL baselines were trained, and there is no comparison with other state-of-the-art methods in molecular optimization. The evaluation metrics used in the experiments are very simple and do not show if the proposed method can optimize any molecular properties or at least retain high binding affinity. The Authors claim that their search strategy separates property optimization from training, but the results of the optimization are not presented. Additionally, all methods were run only once (if I understand correctly), and the results can be hugely impacted by random initialization, especially for online RL methods like the baselines. I would strongly suggest running these methods multiple times and providing confidence intervals for the evaluation metrics.

Author response:
> "Only two RL baselines were trained, and there is no comparison with other state-of-the-art methods in molecular optimization."

We refer the reviewer to Common Response (2).

> "The evaluation metrics used in the experiments are very simple."

We refer the reviewer to Common Response (3).

A table with additional baseline and metrics is present in the "Results Table" section of Common Response.

Reviewer comment (Evaluation):
What is the success rate of molecular optimization using LOGRL? Can you find many well-optimized molecules in the post-training filtering step, or do you think additional RL objectives could improve these properties significantly?

Author response:
> Q1 (A): What is the success rate of molecular optimization using LOGRL? 

Without docking, the notion of success rate depends on the user. For instance, for our chosen examples of trypsin inhibitors, among 10000 molecules generated by LOGRL, 73 had QED > 0.7. 

> Q1 (B): Can you find many well-optimized molecules in the post-training filtering step?

It is possible to find many optimized molecules in the post-training filtering step. Towards this effect, we can simply increase the breadth or depth of the search, which results in a larger number of molecules. 

> Q1 (C): Do you think additional RL objectives could improve these properties significantly?

Additional RL objectives would surely improve the properties of generated molecules. Unfortunately, this has several drawbacks, some are mentioned in our paper. Firstly, for each combination of properties, there would need to be a separate model trained. Secondly, problems related to multi-objective optimization: additional cost of training due to extra hyperparameters, Pareto dominance, conflicting objectives, etc. 

Our method avoids these problems by separating molecular search and property search into two different steps.

Reviewer comment (Experiments):
What are the real-life applications of this optimization algorithm? Can it be used for other optimization problems besides lead optimization (see the problems I mentioned in the "Weaknesses" section)?

Author response:
> Q2: What are the real-life applications of this optimization algorithm? Can it be used for other optimization problems besides lead optimization (see the problems I mentioned in the "Weaknesses" section)?

The proposed algorithm can be used in offline RL settings with a large discrete action space, such as recommender systems and language generation.

Reviewer comment (Experiments):
In Section 3.2, two GCRL methods are described. Did you try the other method and if so, could you provide the comparison results?

Author response:
> Q3: In Section 3.2, two GCRL methods are described. Did you try the other method and if so, could you provide the comparison results?

The two methods described are binary rewards and distance-based rewards. Our proposed method uses binary rewards, while the online RL baselines use distance-based rewards. To our understanding, the comparison requested is present in the paper.

We would also like to clarify that similarity-based rewards are a type of distance-based reward.