Original: Reinforced Genetic Algorithm for Structure-based Drug Design
Revised: Reinforced Genetic Algorithm for Structure-based Drug Design

Original: Abstract
Revised: Abstract

Original: Structure-based drug design (SBDD) aims to discover drug candidates by finding molecules (ligands) that bind tightly to a disease-related protein (targets), which is the primary approach to computer-aided drug discovery.
Revised: Structure-based drug design (SBDD) aims to discover drug candidates by finding molecules (ligands) that bind tightly to a disease-related protein (targets), which is the primary approach to computer-aided drug discovery.

Original: Recently, applying deep generative models for three-dimensional (3D) molecular design conditioned on protein pockets to solve SBDD has attracted much attention, but their formulation as probabilistic modeling often leads to unsatisfactory optimization performance.
Revised: Recently, applying deep generative models for three-dimensional (3D) molecular design conditioned on protein pockets to solve SBDD has attracted much attention, but their formulation as probabilistic modeling often leads to unsatisfactory optimization performance.

Original: On the other hand, traditional combinatorial optimization methods such as genetic algorithms (GA) have demonstrated state-of-the-art performance in various molecular optimization tasks.
Revised: On the other hand, traditional combinatorial optimization methods such as genetic algorithms (GA) have demonstrated state-of-the-art performance in various molecular optimization tasks.

Original: However, they do not utilize protein target structure to inform design steps but rely on a random-walk-like exploration, which leads to unstable performance and no knowledge transfer between different tasks despite the similar binding physics.
Revised: However, they do not utilize protein target structure to inform design steps but rely on a random-walk-like exploration, which leads to unstable performance and no knowledge transfer between different tasks despite the similar binding physics.

Original: To achieve a more stable and efficient SBDD, we propose Reinforced Genetic Algorithm ( RGA ) that uses neural models to prioritize the profitable design steps and suppress random-walk behavior.
Revised: To achieve a more stable and efficient SBDD, we propose Reinforced Genetic Algorithm ( RGA ) that uses neural models to prioritize the profitable design steps and suppress random-walk behavior.

Original: The neural models take the 3D structure of the targets and ligands as inputs and are pre-trained using native complex structures to utilize the knowledge of the shared binding physics from different targets and then fine-tuned during optimization.
Revised: The neural models take the 3D structure of the targets and ligands as inputs and are pre-trained using native complex structures to utilize the knowledge of the shared binding physics from different targets and then fine-tuned during optimization.

Original: We conduct thorough empirical studies on optimizing binding affinity to various disease targets and show that RGA outperforms the baselines in terms of docking scores and is more robust to random initializations.
Revised: We conduct thorough empirical studies on optimizing binding affinity to various disease targets and show that RGA outperforms the baselines in terms of docking scores and is more robust to random initializations.

Original: The ablation study also indicates that the training on different targets helps improve the performance by leveraging the shared underlying physics of the binding processes.
Revised: The ablation study also indicates that the training on different targets helps improve the performance by leveraging the shared underlying physics of the binding processes.

Original: The code is available at https://github.com/futianfan/reinforced-genetic-algorithm .
Revised: The code is available at https://github.com/futianfan/reinforced-genetic-algorithm .

Original:  Introduction
Revised:  Introduction

Original: Rapid drug discovery that requires less time and cost is of significant interest in pharmaceutical science.
Revised: Rapid drug discovery that requires less time and cost is of significant interest in pharmaceutical science.

Original: Structure-based drug design (SBDD) [1] that leverages the three-dimensional (3D) structures of the disease-related proteins to design drug candidates is one primary approach to accelerate the drug discovery processes with physical simulation and data-driven modeling.
Revised: Structure-based drug design (SBDD) [1] that leverages the three-dimensional (3D) structures of the disease-related proteins to design drug candidates is one primary approach to accelerate the drug discovery processes with physical simulation and data-driven modeling.

Original: According to the lock and key model [2], the molecules that bind tighter to a disease target are more likely to expose bioactivity against the disease, which has been verified experimentally [3].
Revised: According to the lock and key model [2], the molecules that bind tighter to a disease target are more likely to expose bioactivity against the disease, which has been verified experimentally [3].

Original: As AlphaFold2 has provided accurate predictions to most human proteins [4, 5], SBDD has a tremendous opportunity to discover new drugs for new targets that we cannot model before [6].
Revised: As AlphaFold2 has provided accurate predictions to most human proteins [4, 5], SBDD has a tremendous opportunity to discover new drugs for new targets that we cannot model before [6].

Original: SBDD could be formulated as an optimization problem where the objective function is the binding affinity estimated by simulations such as docking [2].
Revised: SBDD could be formulated as an optimization problem where the objective function is the binding affinity estimated by simulations such as docking [2].

Original: The most widely used design method is virtual screening, which exhaustively investigates every molecule in a library and ranks them.
Revised: The most widely used design method is virtual screening, which exhaustively investigates every molecule in a library and ranks them.

Original: Lyu et al.
Revised: Lyu et al.

Original: successfully discovered new chemotypes for AmpC β -lactamase and the D 4 dopamine receptor by studying hundreds of millions of molecules with docking simulation [7].
Revised: successfully discovered new chemotypes for AmpC β -lactamase and the D 4 dopamine receptor by studying hundreds of millions of molecules with docking simulation [7].

Original: However, the number of the drug-like molecules is large as estimated to be 10 60 [1], and it is computationally prohibitive to screen all of the possible molecules.
Revised: However, the number of the drug-like molecules is large as estimated to be 10 60 [1], and it is computationally prohibitive to screen all of the possible molecules.

Original: Though machine learning approaches have been developed to accelerate screening [8, 9], it is still challenging to screen large enough chemical space within the foreseeable future.
Revised: Though machine learning approaches have been developed to accelerate screening [8, 9], it is still challenging to screen large enough chemical space within the foreseeable future.

Original: Instead of naively screening a library, designing drug candidates with generative models has been highlighted as a promising strategy, exemplified by [10, 11].
Revised: Instead of naively screening a library, designing drug candidates with generative models has been highlighted as a promising strategy, exemplified by [10, 11].

Original: This class of methods models the problem as the generation of ligands conditioned on the protein pockets.
Revised: This class of methods models the problem as the generation of ligands conditioned on the protein pockets.

Original: However, as generative models are trained to learn the distribution of known active compounds, they tend to produce molecules similar to training data [12], which discourages finding novel molecules and leads to unsatisfactory optimization performance.
Revised: However, as generative models are trained to learn the distribution of known active compounds, they tend to produce molecules similar to training data [12], which discourages finding novel molecules and leads to unsatisfactory optimization performance.

Original: A more straightforward solution is a combinatorial optimization algorithm that searches the implicitly defined discrete chemical space.
Revised: A more straightforward solution is a combinatorial optimization algorithm that searches the implicitly defined discrete chemical space.

Original: As shown in multiple standard molecule optimization benchmarks [13, 14, 15], combinatorial optimization methods, especially genetic algorithms (GA)
Revised: As shown in multiple standard molecule optimization benchmarks [13, 14, 15], combinatorial optimization methods, especially genetic algorithms (GA)

Original: [16, 17], often perform better than deep generative models.
Revised: [16, 17], often perform better than deep generative models.

Original: The key to superior performance is GA’s action definition.
Revised: The key to superior performance is GA’s action definition.

Original: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a.
Revised: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a.

Original: parents) and conducts the crossover between two candidates and mutation from a single candidate to generate new offspring.
Revised: parents) and conducts the crossover between two candidates and mutation from a single candidate to generate new offspring.

Original: These two types of actions, crossover and mutation, enable global and local traversal over the chemical space, allowing a thorough exploration and superior optimization performance.
Revised: These two types of actions, crossover and mutation, enable global and local traversal over the chemical space, allowing a thorough exploration and superior optimization performance.

Original: However, most GAs select mutation and crossover operations randomly [16], leading to significant variance between independent runs.
Revised: However, most GAs select mutation and crossover operations randomly [16], leading to significant variance between independent runs.

Original: Especially in SBDD, when the oracle functions are expensive molecular simulations, it is resource-consuming to ensure stability by running multiple times.
Revised: Especially in SBDD, when the oracle functions are expensive molecular simulations, it is resource-consuming to ensure stability by running multiple times.

Original: Further, most current combinatorial methods are designed for general-purpose molecular optimization and simply use a docking simulation as an oracle.
Revised: Further, most current combinatorial methods are designed for general-purpose molecular optimization and simply use a docking simulation as an oracle.

Original: It is challenging to leverage the structure of proteins in these methods, and we need to start from scratch whenever we change a protein target, even though the physics of ligand-protein interaction is shared.
Revised: It is challenging to leverage the structure of proteins in these methods, and we need to start from scratch whenever we change a protein target, even though the physics of ligand-protein interaction is shared.

Original: Ignoring the shared information across tasks leads to unnecessary exploration steps and, thus, demands for many more oracle calls, which require expensive and unnecessary simulations [18].
Revised: Ignoring the shared information across tasks leads to unnecessary exploration steps and, thus, demands for many more oracle calls, which require expensive and unnecessary simulations [18].

Original: To overcome these issues in the GA method, we propose Reinforced Genetic Algorithm ( RGA ), which attempts to reformulate an evolutionary process as a Markov decision process and uses neural networks to make informed decisions and suppress the random-walk behavior.
Revised: To overcome these issues in the GA method, we propose Reinforced Genetic Algorithm ( RGA ), which attempts to reformulate an evolutionary process as a Markov decision process and uses neural networks to make informed decisions and suppress the random-walk behavior.

Original: Specifically, we utilize an E(3)-equivariant neural network [19] to choose parents and mutation types based on the 3D structure of the ligands and proteins.
Revised: Specifically, we utilize an E(3)-equivariant neural network [19] to choose parents and mutation types based on the 3D structure of the ligands and proteins.

Original: The networks are pre-trained with various native complex structures to utilize the knowledge of the shared binding physics between different targets and then fine-tuned with a reinforcement learning algorithm during optimizations.
Revised: The networks are pre-trained with various native complex structures to utilize the knowledge of the shared binding physics between different targets and then fine-tuned with a reinforcement learning algorithm during optimizations.

Original: We test RGA ’s performance with various disease-related targets, including the main protease of SARS-CoV-2.
Revised: We test RGA ’s performance with various disease-related targets, including the main protease of SARS-CoV-2.

Original: The main contributions of this paper can be summarized as follows:
Revised: The main contributions of this paper can be summarized as follows:

Original: • We propose an evolutionary Markov decision process (EMDP) that reformulates an evolutionary process as a Markov decision process, where the state is a population of molecules instead of a single molecule (Section 3.2).
Revised: • We propose an evolutionary Markov decision process (EMDP) that reformulates an evolutionary process as a Markov decision process, where the state is a population of molecules instead of a single molecule (Section 3.2).

Original: • We show the first successful attempt to use a neural model to guide the crossover and mutation operations in a genetic algorithm to suppress random-walk behavior and explore the chemical space intelligently (Section 3.3).
Revised: • We show the first successful attempt to use a neural model to guide the crossover and mutation operations in a genetic algorithm to suppress random-walk behavior and explore the chemical space intelligently (Section 3.3).

Original: • We present a structure-based de novo drug design algorithm that outperforms baseline methods consistently through thorough empirical studies on optimizing binding affinity by leveraging the underlying binding physics (Section 4).
Revised: • We present a structure-based de novo drug design algorithm that outperforms baseline methods consistently through thorough empirical studies on optimizing binding affinity by leveraging the underlying binding physics (Section 4).

Original:  Related Works
Revised:  Related Works

Original: We will discuss the related works on methods of drug design and discuss the advantage of the proposed method over the existing works.
Revised: We will discuss the related works on methods of drug design and discuss the advantage of the proposed method over the existing works.

Original: General Molecular Design .
Revised: General Molecular Design .

Original: Molecular generation methods offer a promising direction for the automated design of molecules with desired pharmaceutical properties such as synthesis accessibility and drug-likeliness.
Revised: Molecular generation methods offer a promising direction for the automated design of molecules with desired pharmaceutical properties such as synthesis accessibility and drug-likeliness.

Original: Based on how to generate or search molecules, these approaches can be categorized into two types, (1) deep generative models (DGMs) imitate the molecular data distribution, including variational autoencoder (VAE) [20, 21], generative adversarial network (GAN) [22, 23], normalizing flow model [24, 25], energy based model [26]; and (2) combinatorial optimization methods directly search over the discrete chemical space, including genetic algorithm (GA)
Revised: Based on how to generate or search molecules, these approaches can be categorized into two types, (1) deep generative models (DGMs) imitate the molecular data distribution, including variational autoencoder (VAE) [20, 21], generative adversarial network (GAN) [22, 23], normalizing flow model [24, 25], energy based model [26]; and (2) combinatorial optimization methods directly search over the discrete chemical space, including genetic algorithm (GA)

Original: [16, 27, 28], reinforcement learning approaches (RL) [29, 30, 31, 32, 33], Bayesian optimization (BO) [34],
Revised: [16, 27, 28], reinforcement learning approaches (RL) [29, 30, 31, 32, 33], Bayesian optimization (BO) [34],

Original: Markov chain Monte Carlo (MCMC) [35, 36, 37] and gradient ascent [38, 39].
Revised: Markov chain Monte Carlo (MCMC) [35, 36, 37] and gradient ascent [38, 39].

Original: General molecular design algorithms often use general black-box oracle functions, and some are only tested with trivial or self-designed oracles.
Revised: General molecular design algorithms often use general black-box oracle functions, and some are only tested with trivial or self-designed oracles.

Original: For example, using penalized octanol-water partition coefficient (LogP) as the oracle function, it grows monotonically with the number of carbons, and thus there exists a trivial policy to optimize LogP. These oracles do not reflect the challenges of real drug discovery, and those algorithms have limited value for pharmaceutical discovery.
Revised: For example, using penalized octanol-water partition coefficient (LogP) as the oracle function, it grows monotonically with the number of carbons, and thus there exists a trivial policy to optimize LogP. These oracles do not reflect the challenges of real drug discovery, and those algorithms have limited value for pharmaceutical discovery.

Original: Recent works are optimizing docking scores to simulate a more realistic discovery scenario [40, 41, 18, 42], same as our work.
Revised: Recent works are optimizing docking scores to simulate a more realistic discovery scenario [40, 41, 18, 42], same as our work.

Original: However, they are still ignoring the information in the given protein structures that could potentially accelerate the design process.
Revised: However, they are still ignoring the information in the given protein structures that could potentially accelerate the design process.

Original: However, the extension to leveraging the structural knowledge is nontrivial.
Revised: However, the extension to leveraging the structural knowledge is nontrivial.

Original: Structure-based Drug Design .
Revised: Structure-based Drug Design .

Original: Structure-based drug design (SBDD) could utilize the structural information to guide the design of molecules, which are potentially more efficient in drug discovery tasks but poses additional challenges of how to leverage the structures.
Revised: Structure-based drug design (SBDD) could utilize the structural information to guide the design of molecules, which are potentially more efficient in drug discovery tasks but poses additional challenges of how to leverage the structures.

Original: Since early 1990s, various
Revised: Since early 1990s, various

Original: SBDD algorithms have been proposed, mostly based on combinatorial optimization algorithms such as tree search [43, 44, 45] and evolutionary algorithms [46, 47].
Revised: SBDD algorithms have been proposed, mostly based on combinatorial optimization algorithms such as tree search [43, 44, 45] and evolutionary algorithms [46, 47].

Original: Those methods typically optimize the ligands in the pockets according to a physical model characterizing the binding affinity.
Revised: Those methods typically optimize the ligands in the pockets according to a physical model characterizing the binding affinity.

Original: For example, RASSE [43] used a force-field-like scoring function [48] to evaluate the partial solutions within a tree search.
Revised: For example, RASSE [43] used a force-field-like scoring function [48] to evaluate the partial solutions within a tree search.

Original: However, obtaining a fast and accurate model to quantify binding free energy itself is still an unsolved challenge.
Revised: However, obtaining a fast and accurate model to quantify binding free energy itself is still an unsolved challenge.

Original: Recently, generative modeling of 3D molecules conditioned on protein targets is attracting more attention [10, 11].
Revised: Recently, generative modeling of 3D molecules conditioned on protein targets is attracting more attention [10, 11].

Original: Similar to DGMs in general molecular design, those methods learn the atom’s compositional and spatial distribution of native structure of protein-ligand complexes with neural models and design new ligands by complete the complex structure given targets.
Revised: Similar to DGMs in general molecular design, those methods learn the atom’s compositional and spatial distribution of native structure of protein-ligand complexes with neural models and design new ligands by complete the complex structure given targets.

Original: Deep generative models are end-to-end and data-driven thus surpass the necessity of understanding the physics of interaction.
Revised: Deep generative models are end-to-end and data-driven thus surpass the necessity of understanding the physics of interaction.

Original: However, as the training objective is to learn the distribution of known active compounds, the models tend to produce molecules close to the training set [12], which is undesired in terms of patentability and leads to unsatisfactory optimization performance.
Revised: However, as the training objective is to learn the distribution of known active compounds, the models tend to produce molecules close to the training set [12], which is undesired in terms of patentability and leads to unsatisfactory optimization performance.

Original:  Method
Revised:  Method

Original: In this paper, we focus on structure-based drug design.
Revised: In this paper, we focus on structure-based drug design.

Original: The goal is to design drug molecules (a.k.a. ligands) that could bind tightly with the disease-related proteins (a.k.a. targets).
Revised: The goal is to design drug molecules (a.k.a. ligands) that could bind tightly with the disease-related proteins (a.k.a. targets).

Original: Given the 3D structures of the target proteins, including binding site information, docking is a popular computational method for assessing the binding affinity, which can be roughly retrieved as the free energy changes during the binding processes.
Revised: Given the 3D structures of the target proteins, including binding site information, docking is a popular computational method for assessing the binding affinity, which can be roughly retrieved as the free energy changes during the binding processes.

Original: We present a variant of genetic algorithm that is guided by reinforcement learning and a docking oracle.
Revised: We present a variant of genetic algorithm that is guided by reinforcement learning and a docking oracle.

Original: Next, we will first describe the general evolutionary process used in genetic algorithms (Section 3.1); Then we will present how to model this evolutionary process as a Markov decision process (MDP) where RL framework can be constructed (Section 3.2);
Revised: Next, we will first describe the general evolutionary process used in genetic algorithms (Section 3.1); Then we will present how to model this evolutionary process as a Markov decision process (MDP) where RL framework can be constructed (Section 3.2);

Original: After that, we describe the detailed implementation of this MDP framework using multiple policy networks (Section 3.3).
Revised: After that, we describe the detailed implementation of this MDP framework using multiple policy networks (Section 3.3).

Original: 3.1 Evolutionary Process
Revised: 3.1 Evolutionary Process

Original: In this section, we introduce the primary setting of the evolutionary processes.
Revised: In this section, we introduce the primary setting of the evolutionary processes.

Original: With both optimization performance and synthetic accessibility taken into account [49, 14], we follow the action settings in
Revised: With both optimization performance and synthetic accessibility taken into account [49, 14], we follow the action settings in

Original: Autogrow 4.0 [17].
Revised: Autogrow 4.0 [17].

Original: It demonstrated superior performance over other GA variants in the empirical validation of structure-based drug design [17], and its mutation actions originated from chemical reactions so that the designed molecules are more likely to be synthesizable.
Revised: It demonstrated superior performance over other GA variants in the empirical validation of structure-based drug design [17], and its mutation actions originated from chemical reactions so that the designed molecules are more likely to be synthesizable.

Original: Specifically, an evolutionary processes starts by randomly sampling a population of drug candidates from a library.
Revised: Specifically, an evolutionary processes starts by randomly sampling a population of drug candidates from a library.

Original: In each generation (iteration), it carries out (i) crossover between parents selected from the last generation, and (ii) mutation on a single child to obtain the offspring pool.
Revised: In each generation (iteration), it carries out (i) crossover between parents selected from the last generation, and (ii) mutation on a single child to obtain the offspring pool.

Original: An illustration of both crossover and mutation operations is available in Appendix.
Revised: An illustration of both crossover and mutation operations is available in Appendix.

Original: Note that we only adopted the action settings from Autogrow 4.0, without using other tricks such as elitism.
Revised: Note that we only adopted the action settings from Autogrow 4.0, without using other tricks such as elitism.

Original: Crossover , also called recombination, combines the structure of two parents to generate new children.
Revised: Crossover , also called recombination, combines the structure of two parents to generate new children.

Original: Following Autogrow 4.0 [17], we select two parents from the last generation and search for the largest common substructure shared between them.
Revised: Following Autogrow 4.0 [17], we select two parents from the last generation and search for the largest common substructure shared between them.

Original: Then we generate two children by randomly switching their decorating moieties, i.e., the side chains attached to the common substructure.
Revised: Then we generate two children by randomly switching their decorating moieties, i.e., the side chains attached to the common substructure.

Original: Mutation operates on a single parent molecule and modifies its structure slightly.
Revised: Mutation operates on a single parent molecule and modifies its structure slightly.

Original: Following Autogrow 4.0 [17], we adopt transformations based on chemical reactions.
Revised: Following Autogrow 4.0 [17], we adopt transformations based on chemical reactions.

Original: Unlike naively defined atom-editing actions, mutation steps based on chemical reactions could ensure all modification is reasonable in reality, leading to a larger probability of designing synthesizable molecules.
Revised: Unlike naively defined atom-editing actions, mutation steps based on chemical reactions could ensure all modification is reasonable in reality, leading to a larger probability of designing synthesizable molecules.

Original: We included two types of chemical reactions: uni-molecular reactions, which only require one reactant, and bi-molecular reactions, which require two reactants.
Revised: We included two types of chemical reactions: uni-molecular reactions, which only require one reactant, and bi-molecular reactions, which require two reactants.

Original: While uni-molecular reactions could be directly applied to the parent, we sample a purchasable compound to react with the parent when conducting a bi-molecular reaction.
Revised: While uni-molecular reactions could be directly applied to the parent, we sample a purchasable compound to react with the parent when conducting a bi-molecular reaction.

Original: In both cases, the parent serves as one reactant, and we use the main product as the child molecule.
Revised: In both cases, the parent serves as one reactant, and we use the main product as the child molecule.

Original: We use the chemical reactions from [17], which was originally from [47, 50].
Revised: We use the chemical reactions from [17], which was originally from [47, 50].

Original: Evolution .
Revised: Evolution .

Original: At the t -th generation (iteration), given a population of molecules denoted as S ( t ) , we generate an offspring pool denoted as Q ( t ) by applying crossover and mutation operations.
Revised: At the t -th generation (iteration), given a population of molecules denoted as S ( t ) , we generate an offspring pool denoted as Q ( t ) by applying crossover and mutation operations.

Original: Then we filter out the ones with undesirable physical and chemical properties (e.g., poor solubility, high toxicity) in the offspring pool and select the most promising K to form the next generation pool ( S ( t +1) ).
Revised: Then we filter out the ones with undesirable physical and chemical properties (e.g., poor solubility, high toxicity) in the offspring pool and select the most promising K to form the next generation pool ( S ( t +1) ).

Original: 3.2 Evolutionary Markov Decision Process
Revised: 3.2 Evolutionary Markov Decision Process

Original: Next we propose the evolutionary Markov decision process (EMDP) that formulates an evolutionary process of genetic algorithm as a Markov decision process (MDP).
Revised: Next we propose the evolutionary Markov decision process (EMDP) that formulates an evolutionary process of genetic algorithm as a Markov decision process (MDP).

Original: The primary purpose is to utilize reinforcement learning algorithms to train networks to inform the decision steps to replace random selections.
Revised: The primary purpose is to utilize reinforcement learning algorithms to train networks to inform the decision steps to replace random selections.

Original: Taking a generation as a state, Markov property that requires P ( S ( t +1) |S (1) , · · · , S ( t ) )
Revised: Taking a generation as a state, Markov property that requires P ( S ( t +1) |S (1) , · · · , S ( t ) )

Original: =
Revised: =

Original: P ( S ( t +1) |S ( t ) ) is naturally satisfied by the evolutionary process described above, where S ( t ) denotes the state at the t -th generation, which is the population of ligands.
Revised: P ( S ( t +1) |S ( t ) ) is naturally satisfied by the evolutionary process described above, where S ( t ) denotes the state at the t -th generation, which is the population of ligands.

Original: We use X to denote a ligand.
Revised: We use X to denote a ligand.

Original: We elaborate essential components for Markov decision process as follows, and the EMDP pipeline is illustrated in Figure 1.
Revised: We elaborate essential components for Markov decision process as follows, and the EMDP pipeline is illustrated in Figure 1.

Original: State Space .
Revised: State Space .

Original: We define the population at the t -step generation, S ( t ) , in the evolutionary process as the state at the t -step in an EMDP.
Revised: We define the population at the t -step generation, S ( t ) , in the evolutionary process as the state at the t -step in an EMDP.

Original: A state includes population of candidate molecules (i.e., ligand, denoted X ) and their 3D poses docked to the target, fully observable to the RL agent.
Revised: A state includes population of candidate molecules (i.e., ligand, denoted X ) and their 3D poses docked to the target, fully observable to the RL agent.

Original: At the beginning of the EMDP, we randomly select a population of candidate molecules and use docking simulation to yield their 3D poses as the initial state.
Revised: At the beginning of the EMDP, we randomly select a population of candidate molecules and use docking simulation to yield their 3D poses as the initial state.

Original: Action Space .
Revised: Action Space .

Original: The actions in an EMDP are to conduct the two evolutionary steps: crossover and mutation, in a population.
Revised: The actions in an EMDP are to conduct the two evolutionary steps: crossover and mutation, in a population.

Original: For each evolutionary step, we need two actions to conduct it.
Revised: For each evolutionary step, we need two actions to conduct it.

Original: Concretely, crossover ( X parent 1 crossover , X parent 2 crossover crossover −−−−−→ X child 1 crossover , X child 2 crossover ) can be divided to two steps: (1) select the first candidate ligand X parent 1 crossover from the current state (population S ( t ) ); (2) conditioned on the first selected candidate X parent 1 crossover , select the second candidate ligand X parent 2 crossover from the remaining candidate ligand set S ( t ) − { X parent 1 crossover } and apply crossover (Section 3.1) to them.
Revised: Concretely, crossover ( X parent 1 crossover , X parent 2 crossover crossover −−−−−→ X child 1 crossover , X child 2 crossover ) can be divided to two steps: (1) select the first candidate ligand X parent 1 crossover from the current state (population S ( t ) ); (2) conditioned on the first selected candidate X parent 1 crossover , select the second candidate ligand X parent 2 crossover from the remaining candidate ligand set S ( t ) − { X parent 1 crossover } and apply crossover (Section 3.1) to them.

Original: candidate to be mutated from the current state population conditioned candidate
Revised: candidate to be mutated from the current state population conditioned candidate

Original: As applying the crossover and mutation steps are deterministic, the actions in an EMDP focus on selecting parents and mutation types.
Revised: As applying the crossover and mutation steps are deterministic, the actions in an EMDP focus on selecting parents and mutation types.

Original: Upon finish the action, we could obtain offspring pool, Q ( t ) .
Revised: Upon finish the action, we could obtain offspring pool, Q ( t ) .

Original: State Transition Dynamics .
Revised: State Transition Dynamics .

Original: The state transition in an EMDP is identical to the evolution in an evolutionary process.
Revised: The state transition in an EMDP is identical to the evolution in an evolutionary process.

Original: Once we finish the actions and obtain the offspring pool, Q ( t ) =
Revised: Once we finish the actions and obtain the offspring pool, Q ( t ) =

Original: { X child 1 , X child 2 , · · · } , we apply molecular quality filters to filter out the ones unlikely to be drug and then select the most promising K to form the parent set for the next generation ( S ( t +1) ).
Revised: { X child 1 , X child 2 , · · · } , we apply molecular quality filters to filter out the ones unlikely to be drug and then select the most promising K to form the parent set for the next generation ( S ( t +1) ).

Original: Reward .
Revised: Reward .

Original: We define the reward as the binding affinity change (docking score).
Revised: We define the reward as the binding affinity change (docking score).

Original: The actions leading to stronger binding score would be prioritized.
Revised: The actions leading to stronger binding score would be prioritized.

Original: As there is no “episode” concept in an EMDP, we treat every step equally.
Revised: As there is no “episode” concept in an EMDP, we treat every step equally.

Original: 3.3 Target-Ligand Policy Network
Revised: 3.3 Target-Ligand Policy Network

Original: To utilize molecular structures’ translational and rotational invariance, we adopt equivariance neural networks (ENNs)
Revised: To utilize molecular structures’ translational and rotational invariance, we adopt equivariance neural networks (ENNs)

Original: [19] as the target-ligand policy neural networks to select the actions in both mutation and crossover steps.
Revised: [19] as the target-ligand policy neural networks to select the actions in both mutation and crossover steps.

Original: Each ligand has a 3D pose that binds to the target protein, and the complex serves as the input of ENN.
Revised: Each ligand has a 3D pose that binds to the target protein, and the complex serves as the input of ENN.

Original: Specifically, we want to model a 3D graph Y , which can be ligand, target, or target-ligand complex.
Revised: Specifically, we want to model a 3D graph Y , which can be ligand, target, or target-ligand complex.

Original: The input feature can be described as Y =
Revised: The input feature can be described as Y =

Original: ( A , Z ) , where A represents atoms’ categories (the vocabulary set V =
Revised: ( A , Z ) , where A represents atoms’ categories (the vocabulary set V =

Original: { H, C, O, N, · · · } ) and Z represents 3D coordinates of the atoms.
Revised: { H, C, O, N, · · · } ) and Z represents 3D coordinates of the atoms.

Original: Suppose
Revised: Suppose

Original: D ∈ R |V|× d is the embedding matrix of all the categories of atoms in a vocabulary set V , is randomly initialized and learnable, d is the hidden dimension in ENN.
Revised: D ∈ R |V|× d is the embedding matrix of all the categories of atoms in a vocabulary set V , is randomly initialized and learnable, d is the hidden dimension in ENN.

Original: Each kind of atom corresponds to a row in D .
Revised: Each kind of atom corresponds to a row in D .

Original: We suppose there are N atoms, and each atom corresponds to a node in the 3D graph.
Revised: We suppose there are N atoms, and each atom corresponds to a node in the 3D graph.

Original: Node embeddings at the l -th layer are denoted as H ( l ) =
Revised: Node embeddings at the l -th layer are denoted as H ( l ) =

Original: { h ( l ) i } Ni =1 , where l = 0 , 1 , · · · , L , L is number of layers in ENN.
Revised: { h ( l ) i } Ni =1 , where l = 0 , 1 , · · · , L , L is number of layers in ENN.

Original: The initial node embedding h (0) i =
Revised: The initial node embedding h (0) i =

Original: D ⊤ a i ∈ R d embeds the i -th node, where a i is one-hot vector that encode the category of the i -th atom.
Revised: D ⊤ a i ∈ R d embeds the i -th node, where a i is one-hot vector that encode the category of the i -th atom.

Original: Coordinate embeddings at the l -th layer are denoted Z ( l ) =
Revised: Coordinate embeddings at the l -th layer are denoted Z ( l ) =

Original: { z ( l ) i } Ni =1 .
Revised: { z ( l ) i } Ni =1 .

Original: The initial coordinate embeddings Z (0) =
Revised: The initial coordinate embeddings Z (0) =

Original: { z i } Ni =1 are the real 3D coordinates of all the nodes.
Revised: { z i } Ni =1 are the real 3D coordinates of all the nodes.

Original: The following equation defines the feedforward rules of ENN, for i, j = 1 , · · · , N, i ̸
Revised: The following equation defines the feedforward rules of ENN, for i, j = 1 , · · · , N, i ̸

Original: = j, l = 0 , 1 , · · · , L − 1 , we have
Revised: = j, l = 0 , 1 , · · · , L − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: where ⊕ denotes the concatenation of vectors; MLP e ( · ) : R 2 d +1 −→ R d ; MLP x ( · ) : R d −→
Revised: where ⊕ denotes the concatenation of vectors; MLP e ( · ) : R 2 d +1 −→ R d ; MLP x ( · ) : R d −→

Original: R ; MLP h ( · ) : R 2 d −→ R d
Revised: R ; MLP h ( · ) : R 2 d −→ R d

Original: are all two-layer multiple layer perceptrons (MLPs) with Swish activation in the hidden layer [51].
Revised: are all two-layer multiple layer perceptrons (MLPs) with Swish activation in the hidden layer [51].

Original: At the l -th layer, w ( l ) ij represents the message vector for the edge from node i to node j ; v ( l ) i represents the message vector for node i , z ( l ) i is the position embedding for node i ; h ( l ) i is the node embedding for node i .
Revised: At the l -th layer, w ( l ) ij represents the message vector for the edge from node i to node j ; v ( l ) i represents the message vector for node i , z ( l ) i is the position embedding for node i ; h ( l ) i is the node embedding for node i .

Original: H ( L ) =
Revised: H ( L ) =

Original: [ h ( L ) 1 , · · · , h ( L ) N ] are the node embeddings of the
Revised: [ h ( L ) 1 , · · · , h ( L ) N ] are the node embeddings of the

Original: L -th (last) layer.
Revised: L -th (last) layer.

Original: We aggregate them using sum function as readout function to obtain a representation of the 3D graph, denoted h Y .
Revised: We aggregate them using sum function as readout function to obtain a representation of the 3D graph, denoted h Y .

Original: The whole process is written as h Y = ENN ( Y ) .
Revised: The whole process is written as h Y = ENN ( Y ) .

Original: Crossover Policy Network .
Revised: Crossover Policy Network .

Original: We design two policy networks for two corresponding actions in a crossover, as mentioned in Section 3.2.
Revised: We design two policy networks for two corresponding actions in a crossover, as mentioned in Section 3.2.

Original: (1) the first action in crossover operation is to select the first parent ligand X parent 1 crossover from the population S ( t ) .
Revised: (1) the first action in crossover operation is to select the first parent ligand X parent 1 crossover from the population S ( t ) .

Original: Similar to the first action in mutation operation, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as the neural network architecture, the selection probability of the ligand X parent 1 crossover ∈ S ( t ) is p
Revised: Similar to the first action in mutation operation, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as the neural network architecture, the selection probability of the ligand X parent 1 crossover ∈ S ( t ) is p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: crossover ( X
Revised: crossover ( X

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where T and X denotes target and ligand (including 3D pose), respectively, T & X denotes target-ligand complex.
Revised: where T and X denotes target and ligand (including 3D pose), respectively, T & X denotes target-ligand complex.

Original: The second action is to select the second parent ligand conditioned on the first parent ligand selected in the first action.
Revised: The second action is to select the second parent ligand conditioned on the first parent ligand selected in the first action.

Original: Specifically, for ligand in the remaining population set, we concatenate the ENN’s embedding of the target, first parent ligand X parent 1 crossover and the second parent ligand X parent 2 mutation , and feed it into an MLP to estimate a scalar as an unnormalized probability.
Revised: Specifically, for ligand in the remaining population set, we concatenate the ENN’s embedding of the target, first parent ligand X parent 1 crossover and the second parent ligand X parent 2 mutation , and feed it into an MLP to estimate a scalar as an unnormalized probability.

Original: The unnormalized probabilities for all the ligands in the remaining population set are normalized via softmax function, i.e., p (2) crossover ( X parent 2 crossover
Revised: The unnormalized probabilities for all the ligands in the remaining population set are normalized via softmax function, i.e., p (2) crossover ( X parent 2 crossover

Original: | X parent 1 crossover , S ( t ) )
Revised: | X parent 1 crossover , S ( t ) )

Original: =
Revised: =

Original: Softmax (cid:8) MLP ( h T ⊕ h X parent1crossover
Revised: Softmax (cid:8) MLP ( h T ⊕ h X parent1crossover

Original: ⊕ h X parent2crossover ) ,
Revised: ⊕ h X parent2crossover ) ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: X parent2crossover ∈S
Revised: X parent2crossover ∈S

Original: [Equation]
Revised: [Equation]

Original: t ) −{ X
Revised: t ) −{ X

Original: [Equation]
Revised: [Equation]

Original: Given two parents ligands, crossover finds the largest substructure that the two parent compounds share and generates a child by combining their decorating moieties.
Revised: Given two parents ligands, crossover finds the largest substructure that the two parent compounds share and generates a child by combining their decorating moieties.

Original: Thus, the generation of child ligands are deterministic, and the probability of the generated ligands X childcrossover is
Revised: Thus, the generation of child ligands are deterministic, and the probability of the generated ligands X childcrossover is

Original: [Equation]
Revised: [Equation]

Original: Mutation Policy Network .
Revised: Mutation Policy Network .

Original: We design two policy networks for two corresponding actions in mutation, as mentioned in Section 3.2.
Revised: We design two policy networks for two corresponding actions in mutation, as mentioned in Section 3.2.

Original: (1) the first action in mutation operation is to select a candidate ligand to be mutated from population S ( t ) .
Revised: (1) the first action in mutation operation is to select a candidate ligand to be mutated from population S ( t ) .

Original: It models the 3D target-ligand complex to learn if there is improvement space in the current complex.
Revised: It models the 3D target-ligand complex to learn if there is improvement space in the current complex.

Original: Formally, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as neural architecture, the selection probability of the ligand X parentmutation ∈ S ( t ) is
Revised: Formally, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as neural architecture, the selection probability of the ligand X parentmutation ∈ S ( t ) is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , where T & X denotes target-ligand complex, h T & X = ENN ( T & X ) represents the ENN’s embedding of target-ligand complex.
Revised: , where T & X denotes target-ligand complex, h T & X = ENN ( T & X ) represents the ENN’s embedding of target-ligand complex.

Original: The second action is to select the SMARTS reaction from the reaction set conditioned on the selected ligand in the first action.
Revised: The second action is to select the SMARTS reaction from the reaction set conditioned on the selected ligand in the first action.

Original: Specifically, for each reaction, we generate the new ligand X childmutation , then obtain the embedding of target, first ligand X parentmutation and the new ligand X childmutation through ENN, concatenate these three embeddings and feed it into an MLP to estimate a scalar as unnormalized probability.
Revised: Specifically, for each reaction, we generate the new ligand X childmutation , then obtain the embedding of target, first ligand X parentmutation and the new ligand X childmutation through ENN, concatenate these three embeddings and feed it into an MLP to estimate a scalar as unnormalized probability.

Original: The unnormalized probabilities for all the reactions are normalized via softmax function, i.e., p (2) mutation ( ξ | X parentmutation , S ( t ) )
Revised: The unnormalized probabilities for all the reactions are normalized via softmax function, i.e., p (2) mutation ( ξ | X parentmutation , S ( t ) )

Original: =
Revised: =

Original: Softmax (cid:8) MLP ( h T ⊕ h X parentmutation ⊕ h X childmutation ]) , · · · , (cid:9) ξ ∈R , where
Revised: Softmax (cid:8) MLP ( h T ⊕ h X parentmutation ⊕ h X childmutation ]) , · · · , (cid:9) ξ ∈R , where

Original: X parentmutation mutated by ξ −−−−−−−→ X childmutation , R is the reaction set.
Revised: X parentmutation mutated by ξ −−−−−−−→ X childmutation , R is the reaction set.

Original: The probability of the generated ligand X childmutation is p
Revised: The probability of the generated ligand X childmutation is p

Original: [Equation]
Revised: [Equation]

Original: ( X childmutation |S
Revised: ( X childmutation |S

Original: [Equation]
Revised: [Equation]

Original: X parentmutation
Revised: X parentmutation

Original: [Equation]
Revised: [Equation]

Original: S
Revised: S

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Policy Gradient .
Revised: Policy Gradient .

Original: We leverage policy gradient to train the target-ligand policy neural network.
Revised: We leverage policy gradient to train the target-ligand policy neural network.

Original: Specifically, we consider maximizing the expected reward as objective via REINFORCE [29],
Revised: Specifically, we consider maximizing the expected reward as objective via REINFORCE [29],

Original: [Equation]
Revised: [Equation]

Original: where p ( X ) is defined in Equation (2) and (3) for crossover and mutation, respectively.
Revised: where p ( X ) is defined in Equation (2) and (3) for crossover and mutation, respectively.

Original: The whole pipeline is illustrated in Figure 1.
Revised: The whole pipeline is illustrated in Figure 1.

Original: To provide a warm start and leverage the structural information, we pretrain the ENNs on 3D target-ligand binding affinity prediction task, where the inputs are the target-ligand complexes, and the outputs are their binding affinity.
Revised: To provide a warm start and leverage the structural information, we pretrain the ENNs on 3D target-ligand binding affinity prediction task, where the inputs are the target-ligand complexes, and the outputs are their binding affinity.

Original:  Experiment
Revised:  Experiment

Original: In this section, we briefly describe the experimental setup and results.
Revised: In this section, we briefly describe the experimental setup and results.

Original: The Appendix includes more details, including software configuration, implementation details, dataset description & processing, hyperparameter tuning, ablation study, and additional experimental results.
Revised: The Appendix includes more details, including software configuration, implementation details, dataset description & processing, hyperparameter tuning, ablation study, and additional experimental results.

Original: The code is available at https://github.com/futianfan/reinforced-genetic-algorithm .
Revised: The code is available at https://github.com/futianfan/reinforced-genetic-algorithm .

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Docking Simulation .
Revised: Docking Simulation .

Original: We adopt AutoDock Vina [52] to evaluate the binding affinity.
Revised: We adopt AutoDock Vina [52] to evaluate the binding affinity.

Original: The docking score estimated by AutoDock Vina is called Vina score and roughly characterizes the free energy changes of binding processes in kcal/mol.
Revised: The docking score estimated by AutoDock Vina is called Vina score and roughly characterizes the free energy changes of binding processes in kcal/mol.

Original: Thus lower Vina score means a stronger binding affinity between the ligand and target.
Revised: Thus lower Vina score means a stronger binding affinity between the ligand and target.

Original: We picked various disease-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E [53] and the SARS-CoV-2 main protease [54] as targets.
Revised: We picked various disease-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E [53] and the SARS-CoV-2 main protease [54] as targets.

Original: Please see the Appendix for more information.
Revised: Please see the Appendix for more information.

Original: Baselines .
Revised: Baselines .

Original: The baseline methods cover traditional brute-force search methods (Screening), deep generative models (JTVAE and Gen3D), genetic algorithm (GA+D, graph-GA, Autogrow 4.0), reinforcement learning methods (MolDQN, RationaleRL, REINVENT, GEGL), and MCMC method (MARS).
Revised: The baseline methods cover traditional brute-force search methods (Screening), deep generative models (JTVAE and Gen3D), genetic algorithm (GA+D, graph-GA, Autogrow 4.0), reinforcement learning methods (MolDQN, RationaleRL, REINVENT, GEGL), and MCMC method (MARS).

Original: Gen3D and Autogrow 4.0 are structure-based drug design methods, while others are general-purpose molecular design methods.
Revised: Gen3D and Autogrow 4.0 are structure-based drug design methods, while others are general-purpose molecular design methods.

Original: Although methods explicitly utilizing target structures are relatively few, we add general-purpose molecular design methods optimizing the same docking oracle scores as ours, which is a common use case, as baselines [16, 14].
Revised: Although methods explicitly utilizing target structures are relatively few, we add general-purpose molecular design methods optimizing the same docking oracle scores as ours, which is a common use case, as baselines [16, 14].

Original: Concretely, Screening mimics high throughput screening via sampling from ZINC database randomly; JTVAE (Junction Tree Variational
Revised: Concretely, Screening mimics high throughput screening via sampling from ZINC database randomly; JTVAE (Junction Tree Variational

Original: Auto-Encoder)
Revised: Auto-Encoder)

Original: [21] uses a Bayesian optimization on the latent space to indirectly optimize molecules;
Revised: [21] uses a Bayesian optimization on the latent space to indirectly optimize molecules;

Original: Gen3D [10] is an auto-regressive generative model that grows 3D structures atom-wise inside the binding pocket; GA+D [27] represents molecule as SELFIES string [55] and uses genetic algorithm enhanced by a discriminator neural network; Graph-GA [16] conduct genetic algorithm on molecular graph representation; Autogrow 4.0 [17] is the state-of-the-art genetic algorithm in structure-based drug design; MolDQN (Molecule Deep Q-Network) [31] leverages deep Q-value learning to grow molecules atom-wisely; RationaleRL [32] uses rationale (e.g., functional groups or subgraphs) as the building block and a policy gradient method to guide the training of graph neural network-based generator; REINVENT [29] represent molecules as SMILES string and uses policy gradient based reinforcement learning methods to guide the training of the RNN generator; GEGL (genetic expertguided learning)
Revised: Gen3D [10] is an auto-regressive generative model that grows 3D structures atom-wise inside the binding pocket; GA+D [27] represents molecule as SELFIES string [55] and uses genetic algorithm enhanced by a discriminator neural network; Graph-GA [16] conduct genetic algorithm on molecular graph representation; Autogrow 4.0 [17] is the state-of-the-art genetic algorithm in structure-based drug design; MolDQN (Molecule Deep Q-Network) [31] leverages deep Q-value learning to grow molecules atom-wisely; RationaleRL [32] uses rationale (e.g., functional groups or subgraphs) as the building block and a policy gradient method to guide the training of graph neural network-based generator; REINVENT [29] represent molecules as SMILES string and uses policy gradient based reinforcement learning methods to guide the training of the RNN generator; GEGL (genetic expertguided learning)

Original: [33] uses LSTM guided by reinforcement learning to imitate the GA exploration;
Revised: [33] uses LSTM guided by reinforcement learning to imitate the GA exploration;

Original: MARS (Markov Molecule Sampling)
Revised: MARS (Markov Molecule Sampling)

Original: [36] leverages Markov chain Monte Carlo sampling (MCMC) with adaptive proposal and annealing scheme to search chemical space.
Revised: [36] leverages Markov chain Monte Carlo sampling (MCMC) with adaptive proposal and annealing scheme to search chemical space.

Original: To conduct a fair comparison, we limit the number of oracle calls to 1,000 times for each method.
Revised: To conduct a fair comparison, we limit the number of oracle calls to 1,000 times for each method.

Original: All the baselines can be run with one-line code using the software ( https://github.com/wenhao-gao/mol_opt ) in practical molecular optimization benchmark [15].
Revised: All the baselines can be run with one-line code using the software ( https://github.com/wenhao-gao/mol_opt ) in practical molecular optimization benchmark [15].

Original: [Table]
Revised: [Table]

Original: Dataset : we randomly select molecules from ZINC [56] database (around 250 thousands drug-like molecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, GA+D).
Revised: Dataset : we randomly select molecules from ZINC [56] database (around 250 thousands drug-like molecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, GA+D).

Original: ZINC also serves as the training data for pretraining the model in JTVAE, REINVENT, RationaleRL, etc.
Revised: ZINC also serves as the training data for pretraining the model in JTVAE, REINVENT, RationaleRL, etc.

Original: We adopt CrossDocked2020 [57] dataset that contains around 22 million ligand-protein complexes as the training data for pretraining the policy neural networks, as mentioned in Section 3.3.
Revised: We adopt CrossDocked2020 [57] dataset that contains around 22 million ligand-protein complexes as the training data for pretraining the policy neural networks, as mentioned in Section 3.3.

Original: More descriptions are available in Appendix.
Revised: More descriptions are available in Appendix.

Original: Metrics .
Revised: Metrics .

Original: The selection of evaluation metrics follows recent works in molecule optimization [21, 27, 32, 36] and structure-based drug design [17, 10, 14].
Revised: The selection of evaluation metrics follows recent works in molecule optimization [21, 27, 32, 36] and structure-based drug design [17, 10, 14].

Original: For each method, we select top-100 molecules with the best docking scores for evaluation and consider the following metrics: TOP-1/10/ (average docking score of top-1/10/100 molecules):
Revised: For each method, we select top-100 molecules with the best docking scores for evaluation and consider the following metrics: TOP-1/10/ (average docking score of top-1/10/100 molecules):

Original: docking score directly measures the binding affinity between the ligand and target and is the most informative metric in structure-based drug design; Novelty (Nov) (% of the generated molecules that are not in training set); Diversity (Div) (average pairwise Tanimoto distance between the Morgan fingerprints); We also evaluate some simple pharmaceutical properties, including quantitative drug-likeness ( QED ) and synthetic accessibility ( SA ).
Revised: docking score directly measures the binding affinity between the ligand and target and is the most informative metric in structure-based drug design; Novelty (Nov) (% of the generated molecules that are not in training set); Diversity (Div) (average pairwise Tanimoto distance between the Morgan fingerprints); We also evaluate some simple pharmaceutical properties, including quantitative drug-likeness ( QED ) and synthetic accessibility ( SA ).

Original: QED score indicates drug-likeliness ranging from 0 to 1 (higher the better).
Revised: QED score indicates drug-likeliness ranging from 0 to 1 (higher the better).

Original: SA score ranges from 1 to 10 (lower the better).
Revised: SA score ranges from 1 to 10 (lower the better).

Original: All the evaluation functions are available at Therapeutics data commons (TDC, https://tdcommons.ai/fct_overview )
Revised: All the evaluation functions are available at Therapeutics data commons (TDC, https://tdcommons.ai/fct_overview )

Original: 4.2 Results
Revised: 4.2 Results

Original: Stronger Optimization Performance .
Revised: Stronger Optimization Performance .

Original: We summarized the main results of the structure-based drug design in Table 1.
Revised: We summarized the main results of the structure-based drug design in Table 1.

Original: We evaluate all the methods on all targets and report each metric’s mean and standard deviations across all targets.
Revised: We evaluate all the methods on all targets and report each metric’s mean and standard deviations across all targets.

Original: Our result shows RGA achieves the best performance in
Revised: Our result shows RGA achieves the best performance in

Original: TOP-100/10/1 scores among all methods we compared.
Revised: TOP-100/10/1 scores among all methods we compared.

Original: Compared to Autogrow 4.0, RGA ’s better performance in docking score demonstrates that the policy networks contribute positively to the chemical space navigation and eventually help discover more potent binding molecules.
Revised: Compared to Autogrow 4.0, RGA ’s better performance in docking score demonstrates that the policy networks contribute positively to the chemical space navigation and eventually help discover more potent binding molecules.

Original: On the other hand, including longer-range navigation steps enabled by crossover leads to superior performance than other RL methods (REINVENT, MolDQN, GEGL and RationaleRL) that only focus on local modifications.
Revised: On the other hand, including longer-range navigation steps enabled by crossover leads to superior performance than other RL methods (REINVENT, MolDQN, GEGL and RationaleRL) that only focus on local modifications.

Original: In addition, we also observed competitive structure quality measured by QED ( > 0 . 7 ) and SA_Score ( < 2 . 5 ) in Autogrow 4.0 and RGA without involving them as optimization objectives, thanks to the mutation steps originating from chemical reactions.
Revised: In addition, we also observed competitive structure quality measured by QED ( > 0 . 7 ) and SA_Score ( < 2 . 5 ) in Autogrow 4.0 and RGA without involving them as optimization objectives, thanks to the mutation steps originating from chemical reactions.

Original: We visualize two designed ligands with optimal affinity for closer inspection in Figure 2(a) and 2(b), and find both ligands bind tightly with the targets.
Revised: We visualize two designed ligands with optimal affinity for closer inspection in Figure 2(a) and 2(b), and find both ligands bind tightly with the targets.

Original: Suppressed Random-Walk Behavior .
Revised: Suppressed Random-Walk Behavior .

Original: Especially in SBDD, when the oracle functions are expensive molecular simulations, robustness to random seeds is essential for improving the worst-case performance of algorithms.
Revised: Especially in SBDD, when the oracle functions are expensive molecular simulations, robustness to random seeds is essential for improving the worst-case performance of algorithms.

Original: One of the major issues in traditional GAs is that they have a significant variance between multiple independent runs as they randomly select parents for crossover and mutation types.
Revised: One of the major issues in traditional GAs is that they have a significant variance between multiple independent runs as they randomly select parents for crossover and mutation types.

Original: To examine this behavior, we run five independent runs for RGA , Autogrow 4.0 and graph-GA (three best baselines, all are GA methods) on all targets and plot the standard deviations between runs in Figure 2(c) and 2(d).
Revised: To examine this behavior, we run five independent runs for RGA , Autogrow 4.0 and graph-GA (three best baselines, all are GA methods) on all targets and plot the standard deviations between runs in Figure 2(c) and 2(d).

Original: With policy networks guiding the action steps, we observed that the random-walk behavior in Autogrow 4.0 was suppressed in RGA , indicated by the smaller variance.
Revised: With policy networks guiding the action steps, we observed that the random-walk behavior in Autogrow 4.0 was suppressed in RGA , indicated by the smaller variance.

Original: [Figure]
Revised: [Figure]

Original: Especially in the later learning phase (after 500 oracle calls), the policy networks are fine-tuned and guide the search more intelligently.
Revised: Especially in the later learning phase (after 500 oracle calls), the policy networks are fine-tuned and guide the search more intelligently.

Original: This advantage leads to improved worst-case performance and a higher probability of successfully identifying bioactive drug candidates with constrained resources.
Revised: This advantage leads to improved worst-case performance and a higher probability of successfully identifying bioactive drug candidates with constrained resources.

Original: Knowledge Transfer Between Protein Targets .
Revised: Knowledge Transfer Between Protein Targets .

Original: To verify if RGA benefited from learning the shared physics of ligand-target interaction, we conducted an ablation study whose results are in the last two rows of Table 1.
Revised: To verify if RGA benefited from learning the shared physics of ligand-target interaction, we conducted an ablation study whose results are in the last two rows of Table 1.

Original: Specifically, we compare RGA with two variants: (1) RGA -pretrain that does not pretrain the policy network with all native complex structures in the CrossDocked2020; (2) RGA -KT (knowledge transfer) that fine-tune the networks with data of individual target independently.
Revised: Specifically, we compare RGA with two variants: (1) RGA -pretrain that does not pretrain the policy network with all native complex structures in the CrossDocked2020; (2) RGA -KT (knowledge transfer) that fine-tune the networks with data of individual target independently.

Original: We find that both strategies positively contribute to RGA on TOP-100/10/1 docking score.
Revised: We find that both strategies positively contribute to RGA on TOP-100/10/1 docking score.

Original: These results demonstrate the policy networks successfully learn the shared physics of ligand-target interactions and leverage the knowledge to improve their performance.
Revised: These results demonstrate the policy networks successfully learn the shared physics of ligand-target interactions and leverage the knowledge to improve their performance.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose Reinforced Genetic Algorithm ( RGA ) to tackle the structure-based drug design problem.
Revised: In this paper, we propose Reinforced Genetic Algorithm ( RGA ) to tackle the structure-based drug design problem.

Original: RGA reformulate the evolutionary process in genetic algorithms as a Markov decision process called evolutionary Markov decision process (EMDP) so that the searching processes could benefit from trained neural models.
Revised: RGA reformulate the evolutionary process in genetic algorithms as a Markov decision process called evolutionary Markov decision process (EMDP) so that the searching processes could benefit from trained neural models.

Original: Specifically, we train policy networks to choose the parents to crossover and mutate instead of randomly sampling them.
Revised: Specifically, we train policy networks to choose the parents to crossover and mutate instead of randomly sampling them.

Original: Further, we also leverage the common physics of the ligand-target interaction and adopt a knowledge-transfer strategy that uses data from other targets to train the networks.
Revised: Further, we also leverage the common physics of the ligand-target interaction and adopt a knowledge-transfer strategy that uses data from other targets to train the networks.

Original: Through empirical study, we show that RGA has strong and robust optimization performance, consistently outperforming baseline methods in terms of docking score.
Revised: Through empirical study, we show that RGA has strong and robust optimization performance, consistently outperforming baseline methods in terms of docking score.

Original: Though we adopted mutations originating from chemical reactions and the structural quality metrics seem good, we need to emphasize that the designed molecules from RGA do not guarantee synthesizability [49], as the crossover operations may break inheriting synthesizability.
Revised: Though we adopted mutations originating from chemical reactions and the structural quality metrics seem good, we need to emphasize that the designed molecules from RGA do not guarantee synthesizability [49], as the crossover operations may break inheriting synthesizability.

Original: Directly working on synthetic pathways could solve the problem [28, 59], but the extension is not trivial.
Revised: Directly working on synthetic pathways could solve the problem [28, 59], but the extension is not trivial.

Original: As for future direction, we expect to theoretically analyze the EMDP formulation and the performance of RGA .
Revised: As for future direction, we expect to theoretically analyze the EMDP formulation and the performance of RGA .

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: T.F. and J.S. were supported by NSF award SCH-2205289, SCH-2014438, IIS-1838042, NIH award
Revised: T.F. and J.S. were supported by NSF award SCH-2205289, SCH-2014438, IIS-1838042, NIH award

Original: R01 1R01NS107291-01.
Revised: R01 1R01NS107291-01.

Original: W.G. and C.C. were supported by the Office of Naval Research under grant number N00014-21-1-2195 and the Machine Learning for Pharmaceutical Discovery and Synthesis consortium.
Revised: W.G. and C.C. were supported by the Office of Naval Research under grant number N00014-21-1-2195 and the Machine Learning for Pharmaceutical Discovery and Synthesis consortium.

Original: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research
Revised: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A] Contents 1 Introduction 2 Related Works 3 Method 3. Evolutionary Process . .
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: Evolutionary Markov Decision Process . .

Original: Target-Ligand Policy Network . .

Original: 4 Experiment 4.

Original: Experimental Setup . .

Original: Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 5 Conclusion

Original: A Mathematical Notation

Original: B Illustration of Genetic Algorithm

Original: C Baseline Setup

Original: D Additional Experimental Setup

Original: D.1 Docking Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 Dataset . .

Original: D.3 Evaluation metrics . .

Original: E Implementation Details

Original: E

Original: .1 Software/Hardware Configuration

Original: E.2 Hyperparameter Setup .

Original: E.3 Code Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Additional Experiment

Original: F.

Original: Efficiency Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.

Original: Pretraining Equivariant neural network . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.

Original: Additional Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.

Original: Scalability . . . .

Original: F.

Original: Significance Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.

Original: Example of the generated ligand . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A Mathematical Notation

Original: For ease of exposition

Original: [Table]

Original: we list the mathematical notations in Table 2.

Original: All the mathematical notations are divided into three parts: (1) notation for genetic algorithm (Section 3.1); (2) notation for equivariance neural networks (ENN) [19] (Section 3.3); (3) notations for policy network (Section 3.3)

Original: [Table]

Original: [Table]

Original: layer MLP in ENN with Swish activation [51] in hidden layer ⊕ the concatenation of vectors

Original: Z

Original: [Equation]

Original: initial coordinate embeddings, real 3D coordinates of all the nodes.

Original: [Equation]

Original: Node embeddings at the l -th layer

Original: [Equation]

Original: The initial node embedding that embeds the i -th node

Original: [Equation]
Revised: [N/A]

Original: Coordinate embeddings at the l -th layer w

Original: [Equation]

Original: message vector for the edge from node i to node j at l -th layer v

Original: [Equation]

Original: message vector for node i at l -th layer z

Original: [Equation]

Original: i the position embedding for node i at l -th layer h ( l

Original: [Equation]

Original: i the node embedding for node i at l -th layer h

Original: [Equation]

Original: ENN representation of the 3D graph Y (Equation 1) p (1) crossover ( X parent 1 crossover

Original: |S ( t ) ) probability to select the first parent molecule in crossover p (2) crossover ( X parent 2 crossover | X parent 1 crossover , S ( t ) )

Original: probability to select the second parent molecule in crossover p crossover ( X child 1 crossover , X child 2 crossover |S ( t ) ) probability of two generated child molecules in crossover (Eq 2) p (1) mutation ( X parent mutation |S ( t ) ) probability to select the parent molecule in mutation p (2) mutation ( ξ | X parentmutation , S ( t ) ) probability to select the reaction in mutation p mutation ( X childmutation |S ( t ) ) probability of generated child molecule in mutation (Eq 3)

Original: B Illustration of Genetic Algorithm

Original: Figure 3 provides two examples to illustrate crossover and mutation operations in genetic algorithm described in Section 3.1.

Original: Crossover , also called recombination, combines the structure of two parents to generate new children.

Original: Following Autogrow 4.0 [17], as shown in Figure 3(a), we select two parents from the last generation and search for the largest common substructure shared between them.

Original: Then we generate two children by randomly switching their decorating moieties, i.e., the side chains attached to the common substructure.

Original: Mutation operator performs an in silico chemical reaction to generate an altered child compound (i.e., product in the chemical reaction) derived from a parent (reactant in the chemical reaction), as shown in Figure 3(b).

Original: The chemical reaction here contains two reactants, one is parent molecule, another is from reaction set.

Original: The reaction set R is generated via merging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions) [47] and (2) RobustRxn set (58 reactions [50]).

Original: Each reaction ξ in reaction set R contains a SMARTS string based reaction template and a reactant.

Original: It uses

Original: SMARTS reaction template, together with RDKit [60], to perform chemical mutations efficiently.

Original: The process is written as X parentmutation mutated by ξ −−−−−−−→ X childmutation , where ξ is selected reaction (with a reaction template and another reactant).

Original: The ligand to be mutated and the reaction used for mutation are both randomly selected from previous generation and reaction

Original: set R , respectively.

Original: Compared with the mutation operator in conventional GA that randomly flipping an arbitrary bit, the reaction-based mutation enhance synthesizability of the generated molecules [17].

Original: Mutation operator performs an in silico chemical reaction to generate an altered child compound (i.e., product in the chemical reaction) derived from a parent (reactant in the chemical reaction).

Original: The chemical reaction here contains two reactants, one is parent molecule, another is from reaction set.

Original: [Figure]

Original: decorating moieties.

Original: b)

Original: mutation : given a reactant (i.e

Original: , parent), mutation operator uses SMARTS reaction template (

Original: with

Original: another reactant) to performs

Original: an

Original: in silico chemical reaction to generate child compound (i e product).

Original: C Baseline Setup

Original: In this section, we describe the experimental setting for baseline methods.

Original: Most of the settings follow the original papers.

Original: • GA+D

Original: (genetic algorithm enhanced by discriminator neural network)

Original: [27] utilizes SELFIES string as the representation of molecules, thus guaranteeing the 100% chemical validity of the generated molecules.

Original: Following their original paper, the discriminator neural network is a two-layer fully connected neural network with ReLU activation and sigmoid output layer.

Original: The hidden size is 100.

Original: the size of the output layer is 1.

Original: The input feature of discriminator neural network is a vector of chemical and geometrical properties characterizing the molecules.

Original: The population size is set to 300.

Original: Maximal generation number is set to 1000.

Original: The patience is set to 5.

Original: When the property does not improve when the patience exhausts, the process early stops.

Original: We used Adam optimizer with 1e-3 as the initial learning rate.

Original: beta

Original: ( β ) is the weight of discriminator neural network’s score in fitness evaluation, which is used to select most promising molecules in each generation.

Original: We set β = 1 .

Original: • Graph-GA

Original: [16] uses molecular graph to represent molecules and uses crossover and mutation operations to edit the molecular graph.

Original: After tuning, the size of population is set to 120.

Original: The size of offspring is set to 70.

Original: The mutation rate is set to 0.067.

Original: Graph-GA do not have learnable parameters and is easy to implement.

Original: We use the implementation in GuacaMol [13].

Original: • MolDQN (Molecule Deep Q-Networks) [31] uses molecular graph to represent molecules, formulate the molecule optimization process as a Markov Decision Process (MDP) and utilize Deep Q-value learning to optimize it.

Original: It grows molecular graph atom-wise, that is, in each episode, it adds one atom to the partially generated molecular graph.

Original: The reward function is the negative value of the Vina score.

Original: Following the original paper, maximal step in each episode is 40.

Original: Each step calls oracle once.

Original: The discount factor is 0.9.

Original: Deep Q-network is a multilayer perceptron (MLP) whose hidden dimensions are 1024, 512, 128, 32, respectively.

Original: The model size is 6.4 M. The input of the Q-network is the concatenation of the molecule feature (2048-bit Morgan fingerprint, with a radius of 3) and the number of left steps.

Original: Adam is used as an optimizer with 1e-4 as the initial learning rate.

Original: Only rings with a size of 5 and 6 are allowed.

Original: It leverages ϵ -greedy together with randomized value functions (bootstrapped-DQN) as an exploration policy, ϵ is annealed from 1 to 0.01 in a piecewise linear way.

Original: • RationaleRL [32].

Original: The architecture of the generator is a message-passing network (MPN) followed by MLPs applied in breadth-first order.

Original: The generator is pre-trained on general molecules combined with an encoder and then fine-tuned to maximize the reward function using policy gradient.

Original: The encoder and decoder MPNs both have hidden dimensions of 400.

Original: The dimension of the latent variable is 50.

Original: Adam optimizer is used on both pre-training and fine-tuning with initial learning rates of 1e-3, 5e-4, respectively.

Original: The annealing rate is 0.9.

Original: We pre-trained the model with 20 epochs.

Original: • MARS [36] leverage Markov chain Monte Carlo sampling (MCMC) on molecules with an annealing scheme and an adaptive proposal.

Original: The proposal is parameterized by a graph neural network, which is trained on MCMC samples.

Original: We follow most of the settings in the original paper.

Original: The message passing network has six layers, where the node embedding size is set to 64.

Original: Adam is used as an optimizer with 3e-4 initial learning rate.

Original: To generate a basic unit, top-1000 frequent fragments are drawn from ZINC database [56] by enumerating single bonds to break.

Original: During the annealing process, the temperature T = 0 .

Original: ⌊ t/ 5 ⌋ would gradually decrease to 0.

Original: • Autogrow 4.0 [17] is the base model for RGA and have been briefly described in Section 3.1.

Original: The setup of Autogrow is the same as RGA for fair comparison, the only difference is that RGA use policy network to guide the selection of ligands and reaction for crossover and mutation while Autogrow randomly selects them.

Original: The reaction set R is generated via merging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions) [47] and (2) RobustRxn set (58 reactions [50]).

Original: In each generation, It generates 200 offspring (100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vina scores) ones for the next generation.

Original: • Screening exhaustively searches the ZINC database [56] within oracle budget.

Original: It is the traditional high-throughput screening approach.

Original: • REINVENT [29] is a reinforcement learning approach, represent molecule as SMILES string and uses recurrent neural network to model SMILES string.

Original: It pretrains a prior model using molecules on ZINC and finetune the model using the reward function.

Original: It uses REINFORCE to maximize the expected reward function.

Original: The learning rate is set to 0.0005; the batch size is set to 64; The hyperparameter σ weighs the pretrained prior model and the reward function, and is set to 60.

Original: The model size is 16.3M. • JTVAE [21] build a junction tree to represent molecule via using substructure (either ring or atom) to represent molecule.

Original: It uses both molecular graph-leven and junction tree-level encoder and decoder.

Original: The VAE model is pretrained on ZINC databases.

Original: Then Bayesian Optimization is used to optimize the docking score on the continuous latent space.

Original: We use “botorch”, the python’s Bayesian optimization package, to implement the Bayesian

Original: optimization process.

Original: It has 703 substructures in vocabulary, extracted from ZINC.

Original: The hidden size is 450.

Original: The latent size of VAE is set to 56.

Original: The model size is 21.8 M. • Gen3D [10] uses 3D deep generative models and grow the molecule via adding atoms auto-regressively.

Original: It train a universal model for all the targets.

Original: The number of message passing layers in context encoder is 6, and the hidden dimension is 256.

Original: We train the model using the Adam optimizer at learning rate 0.0001.

Original: The model size is 17.4 M. • GEGL (Genetic Expert Guided Learning)

Original: [33] uses LSTM (guided by RL agent) to imitate GA process, however, it is unable to inherit the GA’s flexible assembling manner due to the auto-regressive essence of LSTM.

Original: It use Adam as optimizer with initial learning rate 1e-3.

Original: The batch size during sampling is 512, the batch size during optimization is 256.

Original: In GA, mutation rate is 0.01.

Original: The similarity threshold is 0.4, which constrain the similarity between the original molecule and the edited molecule.

Original: The maximal SMILES length is set to 120.

Original: D Additional Experimental Setup D.1 Docking Simulation

Original: Molecular docking is a computational method which predicts the preferred orientation of one molecule to a second when a ligand and a target are bound to each other to form a stable complex.

Original: Knowledge of the preferred orientation in turn may be used to predict the strength of association or binding affinity between two molecules using, for example, scoring functions.

Original: We adopt AutoDock Vina [52] to evaluate the binding affinity.

Original: The docking score estimated by AutoDock Vina is called Vina score and roughly characterizes the free energy changes of binding processes in kcal/mol.

Original: Vina score is usually smaller than 0 and lower Vina score means a stronger binding affinity between the ligand and target.

Original: We leverage the negative value of the docking score as reward function (Equation 4).

Original: D.2 Dataset

Original: In this paper, we use ZINC [56] database and CrossDocked2020 [57] dataset.

Original: ZINC is a free database of 250 thousands commercially-available drug-like chemical compounds for virtual screening [56].

Original: We randomly select molecules from ZINC [56] database (around 250 thousands drug-like molecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, graph-GA GA+D).

Original: Other baseline methods also use ZINC to either pretrain the models, e.g., JTVAE, REINVENT, RationaleRL or provide searching database, e.g., screening.

Original: We adopt CrossDocked2020 [57] dataset that contains around 22 million ligand-protein complexes as the training data for pretraining the policy neural networks, as mentioned in Section 3.3.

Original: Regarding the target proteins, we picked various disease-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E [53] and the SARS-CoV-2 main protease [54] as targets.

Original: for all the selected target protein, the binding pocket size for all the targets are set to (15.0, 15.0, 15.0).

Original: The units of coordinate are Angstrom ˚ A ( 10 − 10 m).

Original: Detailed descriptions of these targets are available at https://www.rcsb.org/ .

Original: D.3 Evaluation metrics

Original: We leverage the following evaluation metrics to measure the optimization performance:

Original: • Novelty is the fraction of the generated molecules that do not appear in the training set.

Original: • Diversity of generated molecules is defined as the average pairwise Tanimoto distance between the Morgan fingerprints [30, 32, 36].

Original: [Equation]

Original: where M is the set of generated molecules that we want to evaluate.

Original: sim ( m 1 , m 2 ) is the Tanimoto similarity between molecule m 1 and m 2 , where (Tanimoto) Similarity measures the similarity between the input molecule and generated molecules.

Original: It is defined as

Original: [Equation]

Original: , Y )

Original: [Equation]

Original: FP ⊤ X FP Y ∥

Original: FP X ∥ 2 ∥ FP Y ∥ 2 , FP X is the binary Morgan fingerprint vector for the molecule

Original: X .

Original: In this paper, it is a 2048-bit binary vector.

Original: • QED represents a quantitative estimate of drug-likeness.

Original: QED score ranges from 0 to 1.

Original: It can be evaluated by the RDKit package ( https://www.rdkit.org/ ).

Original: • SA (Synthetic Accessibility) score measures how hard it is to synthesize a given molecule, based on a combination of the molecule’s fragments contributions [61].

Original: It is evaluated via RDKit [60].

Original: The raw SA score ranges from 1 to 10.

Original: A higher SA score means the molecule is hard to be synthesized and is not desirable.

Original: • Run Time .

Original: Unlike optimizing some simple oracles such as QED and LogP scores, the docking simulation need to search 3D molecular conformation docked in the target, which is computationally expensive.

Original: Thus run time is an important metric to measure the efficiency of the methods.

Original: E Implementation Details E.1 Software/Hardware Configuration

Original: We implemented RGA using Pytorch 1.10.2, Python 3.7, RDKit v2020.09.1.0 on an Intel Xeon

Original: E5-2690 machine with 256G RAM and NVIDIA Pascal Titan X GPUs.

Original: E.2 Hyperparameter Setup

Original: The neural architectures of policy networks are E(3)-equivariant neural network (ENN) [19].

Original: The vocabulary set V = { C, N, O, S, H, other } .

Original: In ENN, the number of layers is set to 3, i.e., L = 3 ; the hidden dimension is set to 100, i.e., d = 100 .

Original: In Equation 1, MLP e ( · ) , MLP x ( · ) , MLP h ( · ) are two-layer MLP in ENN with Swish activation [51] in hidden layer.

Original: Summation function is used as aggregation function to aggregate the last-layer’s node embedding into graph-level embedding.

Original: All the atoms that within the binding site are used as the input of ENN.

Original: REINFORCE is used to implement policy gradient [62, 29].

Original: Adam is utilized as optimizer with learning rate 0 .

Original: 001 for both crossover and mutation policy networks.

Original: The reaction set R is generated via merging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions) [47] and (2) RobustRxn set ( reactions [50]).

Original: We use RDKit [60] to perform in silico chemical reaction based on SMARTS reaction template, then we have |R| = 36 + 58 = 94 .

Original: In each generation, we generate up to 200 offspring (100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vina scores) ones for the next generation, i.e., K = 50 .

Original: E.3 Code Repository

Original: The code repository is uploaded in supplementary material for reproducibility.

Original: F Additional Experiment F.1 Efficiency Study

Original: As mentioned before, unlike optimizing some simple oracles such as QED and LogP scores, the docking simulation need to search 3D molecular conformation docked in the target, which is timeconsuming.

Original: Thus run time is an important measurement to evaluate the efficiency of the methods.

Original: We report the bar of run time over different targets for all the compared methods in Figure 4.

Original: The run times varies greatly over different methods.

Original: Thus, for ease of visualization, we divided all the methods into two groups.

Original: One is slow group, containing 5 methods: screening, GEGL, REINVENT,

Original: RationaleRL and JTVAE, where all the methods take more than 10 hours.

Original: Another is fast group ( <hours), containing 7 methods, MARS, MolDQN, Gen3D, GA+D, GraphGA, Autogrow, and RGA.

Original: We find that both Autogrow and RGA are efficient compared with other methods.

Original: This attributes to the unique design of genetic algorithm (both crossover and mutation operations) and the usage of filter after GA operators, as described in Section 3.1.

Original: RGA is only slightly slower than Autogrow because it requires additional computation to pretrain/train the policy neural networks.

Original: [Figure]

Original: F.2 Pretraining Equivariant neural network

Original: Pretraining equivariant neural network is a crucial step to RGA .

Original: We adopt CrossDocked2020 [57] dataset that contains around 22 million ligand-protein complexes as the training data for pretraining the policy neural networks, as mentioned in Section 3.3.

Original: We split the whole dataset into training/validation dataset with ratio of 9:1.

Original: Each data point is a target-ligand complex and the binding affinity (scalar).

Original: We report the validation loss of ENN on target-ligand binding affinity prediction task.

Original: The validation loss function is root mean square error (RMSE).

Original: We find the learning process converges rapidly when passing 150K data points (within an epoch) in terms of validation RMSE loss.

Original: F.3 Additional Ablation Study

Original: To further understand our model and GA process, we conduct an ablation study to investigate the impact of each component/strategy to optimization performance.

Original: Specifically, we consider the following four variants of RGA.

Original: RGA -pretrain is a variant of RGA that does not pretrain the policy neural network.

Original: RGA -KT (Knowledge Transfer) is a variant of RGA that does not training policy neural network on different target proteins, i.e., optimizing ligand for one target at a time.

Original: RGA -MU (mutation) is a variant of RGA that does not involve mutation operation in GA.

Original: That is, all the ligands affinity prediction.

Original: We plot the root-mean-square error (RMSE) loss as a function of number of passed data.

Original: We use early stop strategy to terminate the learning process earlier when the validation loss would not decrease to save computational resource and avoid overfitting.

Original: We found the learning process would converges when passing 150K data samples (within an epoch), RMSE loss decreases from more than 8 to less than 1.

Original: [Table]

Original: are generated via crossover operator.

Original: Correspondingly, RGA -CO (crossover) is a variant that does not use crossover operation in GA, which means no mutation operator.

Original: The results are reported in Table 3.

Original: We find that removing either component/strategy will cause a drop in optimization performance (i.e., increase in TOP-100/10/1 scores).

Original: Both crossover and mutation are critical to the optimization performance.

Original: Also, both pretraining the policy networks and knowledge transfer between different target have positive contribution to the performance.

Original: The ablation study furtherly validates the effectiveness of the proposed RGA method.

Original: F.4 Scalability

Original: As mentioned before, the population size in RGA is K .

Original: Then we analyze the computational complexity within each generation.

Original: There are two operations, including crossover and mutation operations, as described in Section 3.3.

Original: For crossover, the first step is to select the first parent molecule, we need to evaluate the probability over all the molecules in current population, whose complexity is O ( K ) .

Original: The second step is to select the second parent molecule based on the first parent molecule, we need to evaluate the probability over all the remaining molecules in current population, as shown in Equation (2), whose complexity is O

Original: ( K − 1) .

Original: The complexity of crossover operation is O ( K ) .

Original: On the other hand, for mutation operation, the first step is to select the parent molecule (only one parent) via evaluating the probability over all the molecules in the current population, i.e., p (1) mutation ( X parentmutation |S ( t ) ) , whose complexity is O ( K ) .

Original: The second step is to select a mutated molecules

Original: [Table]

Original: (generated by chemical reaction), as shown in Equation (3)

Original: The complexity is O ( |R| ) , where R is the reaction set (see Table 2).

Original: The complexity of mutation operation is O ( |R| + K ) .

Original: To summarize

Original: [Table]

Original: the complexity of RGA is O

Original: ( K + |R| ) .

Original: That is, RGA scales linearly in population size

Original: K and the size of the chemical reaction set |R| .

Original: As mentioned in Section E.2, |R| = 94 , K = 50 .

Original: Thus, RGA owns desired scalability and is not computational expensive.

Original: F.5 Significance Studies

Original: In this section, we present the significance studies.

Original: Specifically, we conduct the hypothesis testing to show the statistical significance of our method over the other GA methods.

Original: We compare the significance of the improvement over other GA methods (including AutoGrow4, Graph-GA, GA+D, and GEGL) via running 5 independent trials with different random seeds and then evaluating the p-value .

Original: We consider the top-100 and top-10 score as the major metrics, which are the most important metrics for optimization performance.

Original: We compare RGA versus AutoGrow 4, Graph-GA and GA+D.

Original: The results (p-value) are shown in Table 4.

Original: We find almost all the p-values are less than 0.05 (except one value), which indicates that the improvements of RGA over other GA methods are statistically significant.

Original: F.6 Example of the generated ligand

Original: This section shows some examples of the generated ligands with desirable binding affinity for various target proteins in Figure 6, 7, 8 and 9, respectively.

Original: We observe that the generated ligands bind tightly with the target proteins.

Original: G Additional Discussion on Related Work

Original: Methodology .

Original: The molecule generations methods can be divided into two categories.

Original: The first one is deep generative models (DGMs), which leverage the continuous representation to estimate the data distribution using various kinds of deep neural networks, including variational autoencoder (VAE) [20, 21], generative adversarial network (GAN) [22, 23], normalizing flow model [24, 63, 25], energy based model [26] and diffusion model [64], etc.

Original: The second one is combinatorial optimization methods, which directly search the discrete chemical space, including genetic algorithm (GA) [16, 27, 28], reinforcement learning approaches (RL) [29, 30, 31, 32], Bayesian Optimization (BO) [34, 65],

Original: Monte Carlo Tree Search (MCTS) [66, 67, 32] and Markov Chain Monte Carlo (MCMC) [35, 36, 37].

Original: Deep generative models (DGMs) usually require a large amount of data to fit, which impedes their usage in low data regime.

Original: In contrast, combinatorial optimization methods require less training data, while the trade-off is the need to call the optimization oracles during the exploration in the chemical space [31, 38, 49, 28, 13].

Original: Among all the machine learning methods, Genetic algorithm (GA) exhibits superior performance in some standard benchmarks [13, 14, 15].

Original: The key reason is GA’s global assembling strategy.

Original: Specifically, in each generation (iteration), GA maintains a population of molecule candidates (a.k.a. parents), and conducts the crossover operation between two (random-selected) parent candidates, which enables relatively large exchanges on molecular sub-graph between molecular graphs.

Original: However,

Original: [Figure]

Original: GA is leveraging random-walk based mutation and crossover operations [16, 17] and is essentially based on brute-force

Original: On the other hand, Reinforcement learning (RL) methods are good at navigating the discrete space via prioritizing the promising searching branches and circumventing brute-force search.

Original: The current RLbased methods [29, 30, 31, 32] slightly left behind other state-of-the-art combinatorial optimization methods [38, 14].

Original: The main reason is that current RL based molecule optimization approaches are based on auto-regressive assembling strategy, i.e., growing molecules iteratively via adding a basic building block one time, where the building block can be either a token in SMILES representation [29] or a substructure in molecular graph representation [30, 31, 32].

Original: Such assembling strategy are essentially local search methods, which hinders the algorithm’s ability to overcome the rough optimization landscape (or energy barrier) and is easy to be stuck in the local optimum [68, 69].

Original: [Figure]

Original: Discussion .

Original: Among all the machine learning methods, molecular graph level genetic algorithm (GA) exhibits state-of-the-art performance in some standard molecule optimization benchmarks [13, 14, 15].

Original: The key reason is GA’s assembling manner.

Original: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a. parents), and conducts the crossover operation between two candidates to generate new offspring, which enables thorough exploration to the chemical space.

Original: However, there is still improvement space for GA.

Original: GA are leveraging random-walk based mutation and crossover operations [16] and suffers from brute-force trial and error strategy.

Original: [Figure]

Original: On the other hand, reinforcement learning approaches are good at navigating the discrete space via prioritizing the promising decisions that are worth investigating, for example, AlphaGo successfully applied RL to defeat a professional human Go player [70].

Original: However, the current RL based drug design methods [29, 30, 31] slightly left behind other state-of-the-art combinatorial optimization methods.

Original: The main reason lies at the inferior assembling strategy, which grows molecule in an auto-regressive fashion.

Original: It is hard for this kind of local search strategy to overcome the barrier of the objective, so it is easy to be trapped into the local optimum.

Original: Deep learning methods can also enhance genetic algorithm.

Original: Due to the random selection used in genetic algorithm, it is challenging to apply deep learning methods in the generation of new candidates (molecules in this paper).

Original: Deep learning can be used to compose fitness evaluation to select the offspring.

Original: For example, GA+D

Original: [27] leverages deep neural network as a discriminator to measure the drug’s proximity to the training data, which is incorporated as a scorer in fitness evaluation.

Original: [71] train a deep neural network-based property predictor and leverage it to enhance the evolutionary algorithm.

Original: In this paper, we attempt to enhance genetic algorithm using reinforcement learning technique.

Original: Specifically, we propose Reinforced Genetic Algorithm ( RGA ), which inherits the assembling manner from genetic algorithm and use reinforcement learning to guide the search over the chemical space.

Original: [33] also combine RL and GA, which uses LSTM (guided by RL agent) to imitate GA process, however, it is unable to inherit the GA’s flexible assembling manner due to the auto-regressive essence of LSTM.

Original: [Figure]