RL -Guider: Leveraging Historical Decisions and Feedback for Drug Editing with Large Language Models (2025)

Abstract: Recent success of large language models (LLMs) in diverse domains showcases their potential to revolutionize scientific fields, including drug editing. Traditional drug editing relies on iterative conversations with domain experts, refining the drug until the desired property is achieved. This interactive and iterative process mirrors the strengths of LLMs, making them well-suited for drug editing. *In existing works, LLMs edit each molecule independently without leveraging knowledge from past edits.* However, human experts develop intuition about effective modifications over time through historical experience; accumulating past knowledge is pivotal for human experts, and so it is for LLMs. *In this work, we propose RL-Guider — a reinforcement-learning agent to provide suggestions to LLMs; it uses the rich information provided from evaluating editing results made by the LLM based on the recommendations to improve itself over time.* RL-Guider is the first work that leverages both the comprehensive “world-level” knowledge of LLMs and the knowledge accumulated from historical feedback. As a result, RL-Guider mitigates several shortcomings of existing approaches and demonstrates superior performance. The code is available at [https://github.com/xufliu/RL-Guider](https://github.com/xufliu/RL-Guider).

## introduction the remarkable performance of large language models (llms) across various tasks has recently sparked growing interest in their application to scientific domains @xcite , such as drug editing, which is typically a complex, iterative process that integrates expert knowledge and refinement. drug editing is a specialized task of molecular optimization, aimed at refining drug-like molecules through small, localized structural modifications. typically, it begins with a conversation with domain experts to gather suggestions on modifications. the molecule is then modified accordingly and subjected to computational or experimental property testing. if the revised drug meets the desired criteria, the process concludes; otherwise, the testing results are provided to the domain expert to inform further refinements @xcite @xcite . this iterative refinement process in drug editing perfectly aligns with the greatest strength of llms-their ability to engage in interactive conversations and integrate suggestions. the process of llm-assisted drug editing occurs in iterative rounds, where each cycle refines the molecular structure based on suggestions or guidance. these suggestions can range from simple responses, such as indicating whether the result is incorrect, to more complex guidance incorporating detailed chemical knowledge and specific modification suggestions. as a powerful generative tool with "world-level" knowledge, llms can leverage their vast pre-trained knowledge to propose structurally valid and chemically meaningful modifications. however, their effectiveness heavily depends on the quality of the provided prompts, suggestions, and guidance. hence, the core of llm-assisted drug editing lies in the design of valuable and knowledgeable suggestions to guide the llm to reason and plan @xcite @xcite . existing works mainly focus on leveraging known domain knowledge, such as through a retrieval database @xcite or another llm @xcite , with the assumption that llms store the chemical domain knowledge. however, these approaches can introduce biases that constrain the exploration of novel molecular structures. over-reliance on existing knowledge risks anchoring the drug editing process to feedback user llm agent-free rag (chatdrug) llm planner (chemreasoner) rl-guider (ours) your provided sequence is not correct. give me a new one. @xmath0. we find a sequence @xmath1, which is correct. give me a new one. @xmath2. replace the h with a carboxyl group. give me a new one. @xmath3. add a carboxyl group to the carbon. give me a new one. @xmath4 ## related work machine learning has achieved remarkable success in molecular analysis @xcite @xcite @xcite @xcite @xcite , which provides a powerful tool for drug discovery @xcite . moreover, in recent years, llms @xcite have shown superior performance across diverse nlp tasks including text translation @xcite and editing @xcite . consequently, several studies have explored the application of llms to drug discovery. for example, pioneering research has employed either fine-tuned llms @xcite or llms retrained on existing datasets @xcite to generate potential drug candidates. despite being a crucial task in drug discovery, drug editing remains exceedingly underexplored. compared with discovery, drug editing has stricter requirements for new drugs, including molecular validity, structural similarity, and desired properties, as introduced in sec. relation with prior works. existing works primarily focus on leveraging predefined knowledge, such as retrieval databases @xcite or pretrained knowledgeable llms @xcite . the details of these methods are introduced in appendix a. while these methods can provide domain-specific insights, they may suffer from significant computational costs and potential biases introduced by relying on predefined knowledge. we discuss these shortcomings further in sec. 4.2. we observe that existing works overlook a crucial source of knowledge, leaving it unused in the editing process. in particular, during the iterative editing process, the edited drugs are evaluated to assess their suitability, providing valuable feedback on key aspects such as validity, structural similarity, and chemical properties of the llm-edited drug. this rich information can be used to refine and improve the llm's editing performance over time; however, existing works utilize this feedback merely as a termination or continuation indicator for the iterative process based on whether the results are satisfactory. in our proposed rl-guider, we explicitly utilize this rich information from the feedback as a reward or penalty to train our reinforcement learning agent, allowing it to accumulate knowledge from historical experience and overcome limitations of existing works. ## rl-guider: guidance by historical experience in this section, we introduce rl-guider, our proposed method for providing more effective guidance. in sec. 4.1, we lay out the formulation of rl-guider and in sec. 4.2, we discuss the merits and significant potential it holds. ## experiments in this section, we demonstrate the effectiveness and efficiency of rl-guider through extensive experiments. first, we compare the performance of our rl-guider against several baselines in sec. 5.1. additionally, we conduct ablation studies in sec. 5.2 to evaluate the computational efficiency, learning capability from accumulated historical experience, and hyperparameter sensitivity of rl-guider. ## conclusion and future work in this work, we propose a novel framework, rl-guider, which leverages a reinforcement learning agent to interact with llms and provide valuable suggestions for efficient and effective drug editing. rl-guider is the first work that learns from the rich feedback obtained through the evaluation of candidate edited drugs; as more editing tasks are performed, rl-guider accumulates historical experience and progressively improves its recommendation-making capabilities. rl-guider mitigates several limitations of existing methods, leading to superior performance demonstrated with experiments across various tasks. future work. rl-guider represents a pioneering approach to reinforcement learning-enhanced drug editing with llms, offering vast unexplored potential. for example, exploring different action space designs, including learnable action spaces, could significantly enhance interactions with llms. moreover, rl-guider can be seamlessly integrated with domain-specific models, such as a "knowledge llm", where rl-guider suggests modifications, the llm provides reasoning on why these suggestions could lead to successful edits, and the main llm incorporates both the suggestions and reasoning to perform the editing task.