Original: Reinforced Genetic Algorithm for Structure-based Drug Design
Revised: Reinforced Genetic Algorithm for Structure-based Drug Design

Original: Abstract
Revised: Abstract

Original: Structure-based drug design (SBDD) aims to discover drug candidates by findingmolecules (ligands) that bind tightly to a disease-related protein (targets), which isthe primary approach to computer-aided drug discovery.
Revised: Structure-based drug design (SBDD) aims to discover drug candidates by finding molecules (ligands) that bind tightly to a disease-related protein (targets), which is the primary approach to computer-aided drug discovery.

Original: Recently, applying deepgenerative models for three-dimensional (3D) molecular design conditioned onprotein pockets to solve SBDD has attracted much attention, but their formulationas probabilistic modeling often leads to unsatisfactory optimization performance.
Revised: Recently, applying deep generative models for three-dimensional (3D) molecular design conditioned on protein pockets to solve SBDD has attracted much attention, but their formulation as probabilistic modeling often leads to unsatisfactory optimization performance.

Original: On the other hand, traditional combinatorial optimization methods such as geneticalgorithms (GA) have demonstrated state-of-the-art performance in various molecular optimization tasks.
Revised: On the other hand, traditional combinatorial optimization methods such as genetic algorithms (GA) have demonstrated state-of-the-art performance in various molecular optimization tasks.

Original: However, they do not utilize protein target structure toinform design steps but rely on a random-walk-like exploration, which leads tounstable performance and no knowledge transfer between different tasks despitethe similar binding physics.
Revised: However, they do not utilize protein target structure to inform design steps but rely on a random-walk-like exploration, which leads to unstable performance and no knowledge transfer between different tasks despite the similar binding physics.

Original: To achieve a more stable and efficient SBDD, wepropose Reinforced Genetic Algorithm ( RGA ) that uses neural models to prioritizethe profitable design steps and suppress random-walk behavior.
Revised: To achieve a more stable and efficient SBDD, we propose Reinforced Genetic Algorithm ( RGA ) that uses neural models to prioritize the profitable design steps and suppress random-walk behavior.

Original: The neural modelstake the 3D structure of the targets and ligands as inputs and are pre-trained usingnative complex structures to utilize the knowledge of the shared binding physicsfrom different targets and then fine-tuned during optimization.
Revised: The neural models take the 3D structure of the targets and ligands as inputs and are pre-trained using native complex structures to utilize the knowledge of the shared binding physics from different targets and then fine-tuned during optimization.

Original: We conduct thorough empirical studies on optimizing binding affinity to various disease targetsand show that RGA outperforms the baselines in terms of docking scores and ismore robust to random initializations.
Revised: We conduct thorough empirical studies on optimizing binding affinity to various disease targets and show that RGA outperforms the baselines in terms of docking scores and is more robust to random initializations.

Original: The ablation study also indicates that thetraining on different targets helps improve the performance by leveraging the sharedunderlying physics of the binding processes.
Revised: The ablation study also indicates that the training on different targets helps improve the performance by leveraging the shared underlying physics of the binding processes.

Revised: The code is available at https://github.com/futianfan/reinforced-genetic-algorithm .

Original:  Introduction
Revised:  Introduction

Original: Rapid drug discovery that requires less time and cost is of significant interest in pharmaceuticalscience.
Revised: Rapid drug discovery that requires less time and cost is of significant interest in pharmaceutical science.

Original: Structure-based drug design (SBDD) [1] that leverages the three-dimensional (3D) structuresof the disease-related proteins to design drug candidates is one primary approach to accelerate thedrug discovery processes with physical simulation and data-driven modeling.
Revised: Structure-based drug design (SBDD) [1] that leverages the three-dimensional (3D) structures of the disease-related proteins to design drug candidates is one primary approach to accelerate the drug discovery processes with physical simulation and data-driven modeling.

Original: According to the lockand key model [2], the molecules that bind tighter to a disease target are more likely to exposebioactivity against the disease, which has been verified experimentally [3].
Revised: According to the lock and key model [2], the molecules that bind tighter to a disease target are more likely to expose bioactivity against the disease, which has been verified experimentally [3].

Original: As AlphaFold2 hasprovided accurate predictions to most human proteins [4, 5], SBDD has a tremendous opportunity todiscover new drugs for new targets that we cannot model before [6].
Revised: As AlphaFold2 has provided accurate predictions to most human proteins [4, 5], SBDD has a tremendous opportunity to discover new drugs for new targets that we cannot model before [6].

Original: SBDD could be formulated as an optimization problem where the objective function is the bindingaffinity estimated by simulations such as docking [2].
Revised: SBDD could be formulated as an optimization problem where the objective function is the binding affinity estimated by simulations such as docking [2].

Original: The most widely used design method is virtualscreening, which exhaustively investigates every molecule in a library and ranks them.
Revised: The most widely used design method is virtual screening, which exhaustively investigates every molecule in a library and ranks them.

Original: Lyu et al.successfully discovered new chemotypes for AmpC β -lactamase and the D 4 dopamine receptor bystudying hundreds of millions of molecules with docking simulation [7].
Revised: Lyu et al. successfully discovered new chemotypes for AmpC β -lactamase and the D 4 dopamine receptor by studying hundreds of millions of molecules with docking simulation [7].

Original: However, the number of the drug-like molecules is large as estimated to be 10 60 [1], and it is computationally prohibitive toscreen all of the possible molecules.
Revised: However, the number of the drug-like molecules is large as estimated to be 10 60 [1], and it is computationally prohibitive to screen all of the possible molecules.

Original: Though machine learning approaches have been developed toaccelerate screening [8, 9], it is still challenging to screen large enough chemical space within theforeseeable future.
Revised: Though machine learning approaches have been developed to accelerate screening [8, 9], it is still challenging to screen large enough chemical space within the foreseeable future.

Original: Instead of naively screening a library, designing drug candidates with generative models has beenhighlighted as a promising strategy, exemplified by [10, 11].
Revised: Instead of naively screening a library, designing drug candidates with generative models has been highlighted as a promising strategy, exemplified by [10, 11].

Original: This class of methods models theproblem as the generation of ligands conditioned on the protein pockets.
Revised: This class of methods models the problem as the generation of ligands conditioned on the protein pockets.

Original: However, as generativemodels are trained to learn the distribution of known active compounds, they tend to producemolecules similar to training data [12], which discourages finding novel molecules and leads tounsatisfactory optimization performance.
Revised: However, as generative models are trained to learn the distribution of known active compounds, they tend to produce molecules similar to training data [12], which discourages finding novel molecules and leads to unsatisfactory optimization performance.

Original: A more straightforward solution is a combinatorial optimization algorithm that searches the implicitlydefined discrete chemical space.
Revised: A more straightforward solution is a combinatorial optimization algorithm that searches the implicitly defined discrete chemical space.

Original: As shown in multiple standard molecule optimization benchmarks [13, 14], combinatorial optimization methods, especially genetic algorithms (GA)
Revised: As shown in multiple standard molecule optimization benchmarks [13, 14, 15], combinatorial optimization methods, especially genetic algorithms (GA)

Original: [15, 16],often perform better than deep generative models.
Revised: [16, 17], often perform better than deep generative models.

Original: The key to superior performance is GA’s actiondefinition.
Revised: The key to superior performance is GA’s action definition.

Original: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a.
Revised: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a.

Original: parents) and conducts the crossover between two candidates and mutation from a singlecandidate to generate new offspring.
Revised: parents) and conducts the crossover between two candidates and mutation from a single candidate to generate new offspring.

Original: These two types of actions, crossover and mutation, enableglobal and local traversal over the chemical space, allowing a thorough exploration and superioroptimization performance.
Revised: These two types of actions, crossover and mutation, enable global and local traversal over the chemical space, allowing a thorough exploration and superior optimization performance.

Original: However, most GAs select mutation and crossover operations randomly [15], leading to significantvariance between independent runs.
Revised: However, most GAs select mutation and crossover operations randomly [16], leading to significant variance between independent runs.

Original: Especially in SBDD, when the oracle functions are expensivemolecular simulations, it is resource-consuming to ensure stability by running multiple times.
Revised: Especially in SBDD, when the oracle functions are expensive molecular simulations, it is resource-consuming to ensure stability by running multiple times.

Original: Further,most current combinatorial methods are designed for general-purpose molecular optimization andsimply use a docking simulation as an oracle.
Revised: Further, most current combinatorial methods are designed for general-purpose molecular optimization and simply use a docking simulation as an oracle.

Original: It is challenging to leverage the structure of proteins inthese methods, and we need to start from scratch whenever we change a protein target, even thoughthe physics of ligand-protein interaction is shared.
Revised: It is challenging to leverage the structure of proteins in these methods, and we need to start from scratch whenever we change a protein target, even though the physics of ligand-protein interaction is shared.

Original: Ignoring the shared information across tasksleads to unnecessary exploration steps and, thus, demands for many more oracle calls, which requireexpensive and unnecessary simulations [17].
Revised: Ignoring the shared information across tasks leads to unnecessary exploration steps and, thus, demands for many more oracle calls, which require expensive and unnecessary simulations [18].

Original: To overcome these issues in the GA method, we propose Reinforced Genetic Algorithm ( RGA ),which attempts to reformulate an evolutionary process as a Markov decision process and uses neuralnetworks to make informed decisions and suppress the random-walk behavior.
Revised: To overcome these issues in the GA method, we propose Reinforced Genetic Algorithm ( RGA ), which attempts to reformulate an evolutionary process as a Markov decision process and uses neural networks to make informed decisions and suppress the random-walk behavior.

Original: Specifically, weutilize an E(3)-equivariant neural network [18] to choose parents and mutation types based on the3D structure of the ligands and proteins.
Revised: Specifically, we utilize an E(3)-equivariant neural network [19] to choose parents and mutation types based on the 3D structure of the ligands and proteins.

Original: The networks are pre-trained with various native complexstructures to utilize the knowledge of the shared binding physics between different targets and thenfine-tuned with a reinforcement learning algorithm during optimizations.
Revised: The networks are pre-trained with various native complex structures to utilize the knowledge of the shared binding physics between different targets and then fine-tuned with a reinforcement learning algorithm during optimizations.

Original: We test RGA ’s performancewith various disease-related targets, including the main protease of SARS-CoV-2.
Revised: We test RGA ’s performance with various disease-related targets, including the main protease of SARS-CoV-2.

Original: The main contributions of this paper can be summarized as follows:
Revised: The main contributions of this paper can be summarized as follows:

Original: • We propose an evolutionary Markov decision process (EMDP) that reformulates an evolutionaryprocess as a Markov decision process, where the state is a population of molecules instead of asingle molecule (Section 3.2).• We show the first successful attempt to use a neural model to guide the crossover and mutationoperations in a genetic algorithm to suppress random-walk behavior and explore the chemicalspace intelligently (Section 3.3).• We present a structure-based de novo drug design algorithm that outperforms baseline methodsconsistently through thorough empirical studies on optimizing binding affinity by leveraging theunderlying binding physics (Section 4).
Revised: • We propose an evolutionary Markov decision process (EMDP) that reformulates an evolutionary process as a Markov decision process, where the state is a population of molecules instead of a single molecule (Section 3.2). • We show the first successful attempt to use a neural model to guide the crossover and mutation operations in a genetic algorithm to suppress random-walk behavior and explore the chemical space intelligently (Section 3.3). • We present a structure-based de novo drug design algorithm that outperforms baseline methods consistently through thorough empirical studies on optimizing binding affinity by leveraging the underlying binding physics (Section 4).

Original:  Related Works
Revised:  Related Works

Original: We will discuss the related works on methods of drug design and discuss the advantage of theproposed method over the existing works.
Revised: We will discuss the related works on methods of drug design and discuss the advantage of the proposed method over the existing works.

Original: General Molecular Design .
Revised: General Molecular Design .

Original: Molecular generation methods offer a promising direction for theautomated design of molecules with desired pharmaceutical properties such as synthesis accessibilityand drug-likeliness.
Revised: Molecular generation methods offer a promising direction for the automated design of molecules with desired pharmaceutical properties such as synthesis accessibility and drug-likeliness.

Original: Based on how to generate or search molecules, these approaches can be categorized into two types, (1) deep generative models (DGMs) imitate the molecular data distribution,including variational autoencoder (VAE) [19, 20], generative adversarial network (GAN) [21, 22],normalizing flow model [23, 24], energy based model [25]; and (2) combinatorial optimizationmethods directly search over the discrete chemical space, including genetic algorithm (GA) [15, 26,27], reinforcement learning approaches (RL) [28, 29, 30, 31, 32], Bayesian optimization (BO) [33],
Revised: Based on how to generate or search molecules, these approaches can be categorized into two types, (1) deep generative models (DGMs) imitate the molecular data distribution, including variational autoencoder (VAE) [20, 21], generative adversarial network (GAN) [22, 23], normalizing flow model [24, 25], energy based model [26]; and (2) combinatorial optimization methods directly search over the discrete chemical space, including genetic algorithm (GA) [16, 27, 28], reinforcement learning approaches (RL) [29, 30, 31, 32, 33], Bayesian optimization (BO) [34],

Original: Markov chain Monte Carlo (MCMC) [34, 35, 36] and gradient ascent [37, 38].
Revised: Markov chain Monte Carlo (MCMC) [35, 36, 37] and gradient ascent [38, 39].

Original: General molecular design algorithms often use general black-box oracle functions, and some areonly tested with trivial or self-designed oracles.
Revised: General molecular design algorithms often use general black-box oracle functions, and some are only tested with trivial or self-designed oracles.

Original: For example, using penalized octanol-water partitioncoefficient (LogP) as the oracle function, it grows monotonically with the number of carbons, andthus there exists a trivial policy to optimize LogP. These oracles do not reflect the challenges of realdrug discovery, and those algorithms have limited value for pharmaceutical discovery.
Revised: For example, using penalized octanol-water partition coefficient (LogP) as the oracle function, it grows monotonically with the number of carbons, and thus there exists a trivial policy to optimize LogP. These oracles do not reflect the challenges of real drug discovery, and those algorithms have limited value for pharmaceutical discovery.

Original: Recent worksare optimizing docking scores to simulate a more realistic discovery scenario [39, 40, 17, 41], sameas our work.
Revised: Recent works are optimizing docking scores to simulate a more realistic discovery scenario [40, 41, 18, 42], same as our work.

Original: However, they are still ignoring the information in the given protein structures thatcould potentially accelerate the design process.
Revised: However, they are still ignoring the information in the given protein structures that could potentially accelerate the design process.

Original: However, the extension to leveraging the structuralknowledge is nontrivial.
Revised: However, the extension to leveraging the structural knowledge is nontrivial.

Original: Structure-based Drug Design .
Revised: Structure-based Drug Design .

Original: Structure-based drug design (SBDD) could utilize the structuralinformation to guide the design of molecules, which are potentially more efficient in drug discoverytasks but poses additional challenges of how to leverage the structures.
Revised: Structure-based drug design (SBDD) could utilize the structural information to guide the design of molecules, which are potentially more efficient in drug discovery tasks but poses additional challenges of how to leverage the structures.

Original: Since early 1990s, various
Revised: Since early 1990s, various

Original: SBDD algorithms have been proposed, mostly based on combinatorial optimization algorithms suchas tree search [42, 43, 44] and evolutionary algorithms [45, 46].
Revised: SBDD algorithms have been proposed, mostly based on combinatorial optimization algorithms such as tree search [43, 44, 45] and evolutionary algorithms [46, 47].

Original: Those methods typically optimizethe ligands in the pockets according to a physical model characterizing the binding affinity.
Revised: Those methods typically optimize the ligands in the pockets according to a physical model characterizing the binding affinity.

Original: Forexample, RASSE [42] used a force-field-like scoring function [47] to evaluate the partial solutionswithin a tree search.
Revised: For example, RASSE [43] used a force-field-like scoring function [48] to evaluate the partial solutions within a tree search.

Original: However, obtaining a fast and accurate model to quantify binding free energyitself is still an unsolved challenge.
Revised: However, obtaining a fast and accurate model to quantify binding free energy itself is still an unsolved challenge.

Original: Recently, generative modeling of 3D molecules conditioned on protein targets is attracting moreattention [10, 11].
Revised: Recently, generative modeling of 3D molecules conditioned on protein targets is attracting more attention [10, 11].

Original: Similar to DGMs in general molecular design, those methods learn the atom’scompositional and spatial distribution of native structure of protein-ligand complexes with neuralmodels and design new ligands by complete the complex structure given targets.
Revised: Similar to DGMs in general molecular design, those methods learn the atom’s compositional and spatial distribution of native structure of protein-ligand complexes with neural models and design new ligands by complete the complex structure given targets.

Original: Deep generativemodels are end-to-end and data-driven thus surpass the necessity of understanding the physics ofinteraction.
Revised: Deep generative models are end-to-end and data-driven thus surpass the necessity of understanding the physics of interaction.

Original: However, as the training objective is to learn the distribution of known active compounds,the models tend to produce molecules close to the training set [12], which is undesired in terms ofpatentability and leads to unsatisfactory optimization performance.
Revised: However, as the training objective is to learn the distribution of known active compounds, the models tend to produce molecules close to the training set [12], which is undesired in terms of patentability and leads to unsatisfactory optimization performance.

Original:  Method
Revised:  Method

Original: In this paper, we focus on structure-based drug design.
Revised: In this paper, we focus on structure-based drug design.

Original: The goal is to design drug molecules(a.k.a. ligands) that could bind tightly with the disease-related proteins (a.k.a. targets).
Revised: The goal is to design drug molecules (a.k.a. ligands) that could bind tightly with the disease-related proteins (a.k.a. targets).

Original: Giventhe 3D structures of the target proteins, including binding site information, docking is a popularcomputational method for assessing the binding affinity, which can be roughly retrieved as the freeenergy changes during the binding processes.
Revised: Given the 3D structures of the target proteins, including binding site information, docking is a popular computational method for assessing the binding affinity, which can be roughly retrieved as the free energy changes during the binding processes.

Original: We present a variant of genetic algorithm that is guidedby reinforcement learning and a docking oracle.
Revised: We present a variant of genetic algorithm that is guided by reinforcement learning and a docking oracle.

Original: Next, we will first describe the general evolutionaryprocess used in genetic algorithms (Section 3.1); Then we will present how to model this evolutionaryprocess as a Markov decision process (MDP) where RL framework can be constructed (Section 3.2);
Revised: Next, we will first describe the general evolutionary process used in genetic algorithms (Section 3.1); Then we will present how to model this evolutionary process as a Markov decision process (MDP) where RL framework can be constructed (Section 3.2);

Original: After that, we describe the detailed implementation of this MDP framework using multiple policynetworks (Section 3.3).
Revised: After that, we describe the detailed implementation of this MDP framework using multiple policy networks (Section 3.3).

Original: 3.1 Evolutionary Process
Revised: 3.1 Evolutionary Process

Original: In this section, we introduce the primary setting of the evolutionary processes.
Revised: In this section, we introduce the primary setting of the evolutionary processes.

Original: With both optimizationperformance and synthetic accessibility taken into account [48, 14], we follow the action settings in
Revised: With both optimization performance and synthetic accessibility taken into account [49, 14], we follow the action settings in

Original: Autogrow 4.0 [16].
Revised: Autogrow 4.0 [17].

Original: It demonstrated superior performance over other GA variants in the empiricalvalidation of structure-based drug design [16], and its mutation actions originated from chemicalreactions so that the designed molecules are more likely to be synthesizable.
Revised: It demonstrated superior performance over other GA variants in the empirical validation of structure-based drug design [17], and its mutation actions originated from chemical reactions so that the designed molecules are more likely to be synthesizable.

Original: Specifically, anevolutionary processes starts by randomly sampling a population of drug candidates from a library.
Revised: Specifically, an evolutionary processes starts by randomly sampling a population of drug candidates from a library.

Original: In each generation (iteration), it carries out (i) crossover between parents selected from the lastgeneration, and (ii) mutation on a single child to obtain the offspring pool.
Revised: In each generation (iteration), it carries out (i) crossover between parents selected from the last generation, and (ii) mutation on a single child to obtain the offspring pool.

Original: An illustration of both crossover and mutation operations is available in Appendix.
Revised: An illustration of both crossover and mutation operations is available in Appendix.

Original: Note that we only adopted the actionsettings from Autogrow 4.0, without using other tricks such as elitism.
Revised: Note that we only adopted the action settings from Autogrow 4.0, without using other tricks such as elitism.

Original: Crossover , also called recombination, combines the structure of two parents to generate new children.
Revised: Crossover , also called recombination, combines the structure of two parents to generate new children.

Original: Following Autogrow 4.0 [16], we select two parents from the last generation and search for the largestcommon substructure shared between them.
Revised: Following Autogrow 4.0 [17], we select two parents from the last generation and search for the largest common substructure shared between them.

Original: Then we generate two children by randomly switchingtheir decorating moieties, i.e., the side chains attached to the common substructure.
Revised: Then we generate two children by randomly switching their decorating moieties, i.e., the side chains attached to the common substructure.

Original: Mutation operates on a single parent molecule and modifies its structure slightly.
Revised: Mutation operates on a single parent molecule and modifies its structure slightly.

Original: Following Autogrow4.0 [16], we adopt transformations based on chemical reactions.
Revised: Following Autogrow 4.0 [17], we adopt transformations based on chemical reactions.

Original: Unlike naively defined atom-editingactions, mutation steps based on chemical reactions could ensure all modification is reasonable inreality, leading to a larger probability of designing synthesizable molecules.
Revised: Unlike naively defined atom-editing actions, mutation steps based on chemical reactions could ensure all modification is reasonable in reality, leading to a larger probability of designing synthesizable molecules.

Original: We included two typesof chemical reactions: uni-molecular reactions, which only require one reactant, and bi-molecularreactions, which require two reactants.
Revised: We included two types of chemical reactions: uni-molecular reactions, which only require one reactant, and bi-molecular reactions, which require two reactants.

Original: While uni-molecular reactions could be directly applied to theparent, we sample a purchasable compound to react with the parent when conducting a bi-molecularreaction.
Revised: While uni-molecular reactions could be directly applied to the parent, we sample a purchasable compound to react with the parent when conducting a bi-molecular reaction.

Original: In both cases, the parent serves as one reactant, and we use the main product as the childmolecule.
Revised: In both cases, the parent serves as one reactant, and we use the main product as the child molecule.

Original: We use the chemical reactions from [16], which was originally from [46, 49].
Revised: We use the chemical reactions from [17], which was originally from [47, 50].

Original: Evolution .
Revised: Evolution .

Original: At the t -th generation (iteration), given a population of molecules denoted as S ( t ) , wegenerate an offspring pool denoted as Q ( t ) by applying crossover and mutation operations.
Revised: At the t -th generation (iteration), given a population of molecules denoted as S ( t ) , we generate an offspring pool denoted as Q ( t ) by applying crossover and mutation operations.

Original: Thenwe filter out the ones with undesirable physical and chemical properties (e.g., poor solubility, hightoxicity) in the offspring pool and select the most promising K to form the next generation pool( S ( t +1) ).
Revised: Then we filter out the ones with undesirable physical and chemical properties (e.g., poor solubility, high toxicity) in the offspring pool and select the most promising K to form the next generation pool ( S ( t +1) ).

Original: 3.2 Evolutionary Markov Decision Process
Revised: 3.2 Evolutionary Markov Decision Process

Original: Next we propose the evolutionary Markov decision process (EMDP) that formulates an evolutionaryprocess of genetic algorithm as a Markov decision process (MDP).
Revised: Next we propose the evolutionary Markov decision process (EMDP) that formulates an evolutionary process of genetic algorithm as a Markov decision process (MDP).

Original: The primary purpose is to utilizereinforcement learning algorithms to train networks to inform the decision steps to replace randomselections.
Revised: The primary purpose is to utilize reinforcement learning algorithms to train networks to inform the decision steps to replace random selections.

Original: Taking a generation as a state, Markov property that requires P ( S ( t +1) |S (1) , · · · , S ( t ) ) =
Revised: Taking a generation as a state, Markov property that requires P ( S ( t +1) |S (1) , · · · , S ( t ) ) =

Original: P ( S ( t +1) |S ( t ) ) is naturally satisfied by the evolutionary process described above, where S ( t ) denotesthe state at the t -th generation, which is the population of ligands.
Revised: P ( S ( t +1) |S ( t ) ) is naturally satisfied by the evolutionary process described above, where S ( t ) denotes the state at the t -th generation, which is the population of ligands.

Original: We use X to denote a ligand.
Revised: We use X to denote a ligand.

Original: Weelaborate essential components for Markov decision process as follows, and the EMDP pipeline isillustrated in Figure 1.
Revised: We elaborate essential components for Markov decision process as follows, and the EMDP pipeline is illustrated in Figure 1.

Original: State Space .
Revised: State Space .

Original: We define the population at the t -step generation, S ( t ) , in the evolutionary process asthe state at the t -step in an EMDP.
Revised: We define the population at the t -step generation, S ( t ) , in the evolutionary process as the state at the t -step in an EMDP.

Original: A state includes population of candidate molecules (i.e., ligand,denoted X ) and their 3D poses docked to the target, fully observable to the RL agent.
Revised: A state includes population of candidate molecules (i.e., ligand, denoted X ) and their 3D poses docked to the target, fully observable to the RL agent.

Original: At the beginningof the EMDP, we randomly select a population of candidate molecules and use docking simulation toyield their 3D poses as the initial state.
Revised: At the beginning of the EMDP, we randomly select a population of candidate molecules and use docking simulation to yield their 3D poses as the initial state.

Original: Action Space .
Revised: Action Space .

Original: The actions in an EMDP are to conduct the two evolutionary steps: crossover andmutation, in a population.
Revised: The actions in an EMDP are to conduct the two evolutionary steps: crossover and mutation, in a population.

Original: For each evolutionary step, we need two actions to conduct it.
Revised: For each evolutionary step, we need two actions to conduct it.

Original: Concretely,crossover ( X parent 1 crossover , X parent 2 crossover crossover −−−−−→ X child 1 crossover , X child 2 crossover ) can be divided to: 1. select the first candidate ligand X parent 1 crossover from the current state (population S ( t ) ); 2. conditioned on the first selected candidate X parent 1 crossover , select the second candidate ligand 
Revised: Concretely, crossover ( X parent 1 crossover , X parent 2 crossover crossover −−−−−→ X child 1 crossover , X child 2 crossover ) can be divided to two steps: (1) select the first candidate ligand X parent 1 crossover from the current state (population S ( t ) ); (2) conditioned on the first selected candidate X parent 1 crossover , select the second candidate ligand X parent 2 crossover from the remaining candidate ligand set S ( t ) − { X parent 1 crossover } and apply crossover (Section 3.1) to them.

Original: X parent 2 crossover from

Original: the remaining candidate ligand set S (

Original: t )

Original: − { X parent

Original: 1 crossover } and

Original: apply crossover Section 3. ) to them. Mutation ( X parentmutation mutated by ξ −−−−−−−→ X childmutation ) can be divided to:
Revised: Mutation ( X parent mutation mutated by ξ −−−−−−−→ X childmutation ) can be divided to two steps: (1) select the candidate ligand

Original: 1. select the candidate ligand X parentmutated to be mutated from the current state (population S ( t ) ); 2. conditioned on the selected candidate ligand X parentmutated , select the reaction ξ from the reaction set R and apply it to X parentmutated .
Revised: X parentmutated to be mutated from the current state (population S ( t ) ); (2) conditioned on the selected candidate ligand X parentmutated , select the reaction ξ from the reaction set R and apply it to X parentmutated .

Original: As applying the crossover and mutation steps are deterministic, the actions in an EMDP focus onselecting parents and mutation types.
Revised: As applying the crossover and mutation steps are deterministic, the actions in an EMDP focus on selecting parents and mutation types.

Original: Upon finish the action, we could obtainthe offspring pool, Q ( t ) .
Revised: Upon finish the action, we could obtain offspring pool, Q ( t ) .

Original: State Transition Dynamics .
Revised: State Transition Dynamics .

Original: The state transition in an EMDP is identical to the evolution inan evolutionary process.
Revised: The state transition in an EMDP is identical to the evolution in an evolutionary process.

Original: Once we finish the actions and obtain the offspring pool, Q ( t ) =
Revised: Once we finish the actions and obtain the offspring pool, Q ( t ) =

Original: { X child 1 , X child 2 , · · · } , we apply molecular quality filters to filter out the ones unlikely to be drugand then select the most promising K to form the parent set for the next generation ( S ( t +1) ).
Revised: { X child 1 , X child 2 , · · · } , we apply molecular quality filters to filter out the ones unlikely to be drug and then select the most promising K to form the parent set for the next generation ( S ( t +1) ).

Original: Reward .
Revised: Reward .

Original: We define the reward as the binding affinity change (docking score).
Revised: We define the reward as the binding affinity change (docking score).

Original: The actions leading tostronger binding score would be prioritized.
Revised: The actions leading to stronger binding score would be prioritized.

Original: As there is no “episode” concept in an EMDP, we treatevery step equally.
Revised: As there is no “episode” concept in an EMDP, we treat every step equally.

Original: 3.3 Target-Ligand Policy Network
Revised: 3.3 Target-Ligand Policy Network

Original: To utilize molecular structures’ translational and rotational invariance, we adopt equivariance neuralnetworks (ENNs)
Revised: To utilize molecular structures’ translational and rotational invariance, we adopt equivariance neural networks (ENNs)

Original: [18] as the target-ligand policy neural networks to select the actions in both mutationand crossover steps.
Revised: [19] as the target-ligand policy neural networks to select the actions in both mutation and crossover steps.

Original: Each ligand has a 3D pose that binds to the target protein, and the complexserves as the input of ENN.
Revised: Each ligand has a 3D pose that binds to the target protein, and the complex serves as the input of ENN.

Original: Specifically, we want to model a 3D graph Y , which can be ligand, target, or target-ligand complex.
Revised: Specifically, we want to model a 3D graph Y , which can be ligand, target, or target-ligand complex.

Original: The input feature can be described as Y =
Revised: The input feature can be described as Y =

Original: ( A , Z ) , where A represents atoms’ categories (thevocabulary set V =
Revised: ( A , Z ) , where A represents atoms’ categories (the vocabulary set V =

Original: { H, C, O, N, · · · } ) and Z represents 3D coordinates of the atoms.
Revised: { H, C, O, N, · · · } ) and Z represents 3D coordinates of the atoms.

Original: Suppose
Revised: Suppose

Original: D ∈ R |V|× d is the embedding matrix of all the categories of atoms in a vocabulary set V , is randomlyinitialized and learnable, d is the hidden dimension in ENN.
Revised: D ∈ R |V|× d is the embedding matrix of all the categories of atoms in a vocabulary set V , is randomly initialized and learnable, d is the hidden dimension in ENN.

Original: Each kind of atom corresponds to a rowin D .
Revised: Each kind of atom corresponds to a row in D .

Original: We suppose there are N atoms, and each atom corresponds to a node in the 3D graph.
Revised: We suppose there are N atoms, and each atom corresponds to a node in the 3D graph.

Original: Nodeembeddings at the l -th layer are denoted as H ( l ) =
Revised: Node embeddings at the l -th layer are denoted as H ( l ) =

Original: { h ( l ) i } N i =1 , where l = 0 , 1 , · · · , L , L is numberof layers in ENN.
Revised: { h ( l ) i } N i =1 , where l = 0 , 1 , · · · , L , L is number of layers in ENN.

Original: The initial node embedding h (0) i =
Revised: The initial node embedding h (0) i =

Original: D ⊤ a i ∈ R d embeds the i -th node, wherea i is one-hot vector that encode the category of the i -th atom.
Revised: D ⊤ a i ∈ R d embeds the i -th node, where a i is one-hot vector that encode the category of the i -th atom.

Original: Coordinate embeddings at the l -thlayer are denoted Z ( l ) =
Revised: Coordinate embeddings at the l -th layer are denoted Z ( l ) =

Original: { z ( l ) i } Ni =1 .
Revised: { z ( l ) i } Ni =1 .

Original: The initial coordinate embeddings Z (0) =
Revised: The initial coordinate embeddings Z (0) =

Original: { z i } Ni =1 are the real3D coordinates of all the nodes.
Revised: { z i } Ni =1 are the real 3D coordinates of all the nodes.

Original: The following equation defines the feedforward rules of ENN, for i, j = 1 , · · · , N, i ̸
Revised: The following equation defines the feedforward rules of ENN, for i, j = 1 , · · · , N, i ̸

Original: = j, l = 0 , 1 , · · · , L − 1 , we have[Equation] l
Revised: = j, l = 0 , 1 , · · · , L − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: z ( l +1)
Revised: z ( l +1)

Original: [Equation]
Revised: [Equation]

Original: where ⊕ denotes the concatenation of vectors; MLP e ( · ) : R 2 d +1 −→ R d ; MLP x ( · ) : R d −→
Revised: where ⊕ denotes the concatenation of vectors; MLP e ( · ) : R 2 d +1 −→ R d ; MLP x ( · ) : R d −→

Original: R ; MLP h ( · ) : R 2 d −→ R d
Revised: R ; MLP h ( · ) : R 2 d −→ R d

Original: are all two-layer multiple layer perceptrons (MLPs) with Swish activation in the hidden layer [50].
Revised: are all two-layer multiple layer perceptrons (MLPs) with Swish activation in the hidden layer [51].

Original: At the l -th layer, w ( l ) ij represents the message vector for the edge fromnode i to node j ; v ( l ) i represents the message vector for node i , z ( l ) i is the position embedding for nodei ; h ( l ) i is the node embedding for node i .
Revised: At the l -th layer, w ( l ) ij represents the message vector for the edge from node i to node j ; v ( l ) i represents the message vector for node i , z ( l ) i is the position embedding for node i ; h ( l ) i is the node embedding for node i .

Original: H ( L ) =
Revised: H ( L ) =

Original: [ h ( L ) 1 , · · · , h ( L ) N ] are the node embeddings of the
Revised: [ h ( L ) 1 , · · · , h ( L ) N ] are the node embeddings of the

Original: L -th (last) layer.
Revised: L -th (last) layer.

Original: We aggregate them using sum function as readout function to obtain a representationof the 3D graph, denoted h Y .
Revised: We aggregate them using sum function as readout function to obtain a representation of the 3D graph, denoted h Y .

Original: The whole process is written as h Y = ENN ( Y ) .
Revised: The whole process is written as h Y = ENN ( Y ) .

Original: Crossover Policy Network .
Revised: Crossover Policy Network .

Original: We design two policy networks for two corresponding actions in acrossover, as mentioned in Section 3.2.
Revised: We design two policy networks for two corresponding actions in a crossover, as mentioned in Section 3.2.

Original: (1) the first action in crossover operation is to select thefirst parent ligand X parent 1 crossover from the population S ( t ) .
Revised: (1) the first action in crossover operation is to select the first parent ligand X parent 1 crossover from the population S ( t ) .

Original: Similar to the first action in mutation operation, we obtain a valid probability distribution over all the available ligands based on target-ligandcomplex as input feature and ENN as the neural network architecture, the selection probabilityof the ligand X parent 1 crossover ∈ S ( t ) is p (1) crossover (
Revised: Similar to the first action in mutation operation, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as the neural network architecture, the selection probability of the ligand X parent 1 crossover ∈ S ( t ) is p (1) crossover (

Original: X parent 1 crossover |S ( t ) )
Revised: X parent 1 crossover |S ( t ) )

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , where Tand X denotes target and ligand (including 3D pose), respectively, T & X denotes target-ligandcomplex.
Revised: , where T and X denotes target and ligand (including 3D pose), respectively, T & X denotes target-ligand complex.

Original: The second action is to select the second parent ligand conditioned on the first parent ligand selected in the first action.
Revised: The second action is to select the second parent ligand conditioned on the first parent ligand selected in the first action.

Original: Specifically, for ligand in the remaining population set, weconcatenate the ENN’s embedding of the target, first parent ligand X parent 1 crossover and the second parent ligand X parent 2 mutation , and feed it into an MLP to estimate a scalar as an unnormalized probability.
Revised: Specifically, for ligand in the remaining population set, we concatenate the ENN’s embedding of the target, first parent ligand X parent 1 crossover and the second parent ligand X parent 2 mutation , and feed it into an MLP to estimate a scalar as an unnormalized probability.

Original: The unnormalized probabilities for all the ligands in the remaining population set are normalizedvia softmax function, i.e., p (2) crossover ( X parent 2 crossover | X parent 1 crossover , S ( t ) )
Revised: The unnormalized probabilities for all the ligands in the remaining population set are normalized via softmax function, i.e., p (2) crossover ( X parent 2 crossover | X parent 1 crossover , S ( t ) )

Original: =
Revised: =

Original: Softmax (cid:8) MLP ( h T ⊕ h X parent1crossover ⊕h X parent2crossover ) , · · · , (cid:9)
Revised: Softmax (cid:8) MLP ( h T ⊕ h X parent1crossover ⊕ h X parent2crossover ) , · · · , (cid:9)

Original: X parent2crossover ∈S ( t ) −{ X parent1crossover } .
Revised: X parent2crossover ∈S ( t ) −{ X parent1crossover } .

Original: Given two parents ligands, crossover finds the largest substructure that the two parent compounds share and generates a child by combining their decoratingmoieties.
Revised: Given two parents ligands, crossover finds the largest substructure that the two parent compounds share and generates a child by combining their decorating moieties.

Original: Thus, the generation of child ligands are deterministic, and the probability of the generatedligands X childcrossover is[Equation] X
Revised: Thus, the generation of child ligands are deterministic, and the probability of the generated ligands X childcrossover is

Original: [Equation]
Revised: [Equation]

Original: Mutation Policy Network .
Revised: Mutation Policy Network .

Original: We design two policy networks for two corresponding actions inmutation, as mentioned in Section 3.2.
Revised: We design two policy networks for two corresponding actions in mutation, as mentioned in Section 3.2.

Original: (1) the first action in mutation operation is to select acandidate ligand to be mutated from population S ( t ) .
Revised: (1) the first action in mutation operation is to select a candidate ligand to be mutated from population S ( t ) .

Original: It models the 3D target-ligand complex tolearn if there is improvement space in the current complex.
Revised: It models the 3D target-ligand complex to learn if there is improvement space in the current complex.

Original: Formally, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as neural architecture, the selection probability of the ligand X parentmutation ∈ S ( t ) isp
Revised: Formally, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as neural architecture, the selection probability of the ligand X parentmutation ∈ S ( t ) is p

Original: [Equation]
Revised: [Equation]

Original: Xparentmutation [Equation]
Revised: X

Original: [Equation]
Revised: [Equation]

Original: , where T & X denotes target-ligand complex,h T & X = ENN ( T & X ) represents the ENN’s embedding of target-ligand complex.
Revised: , where T & X denotes target-ligand complex, h T & X = ENN ( T & X ) represents the ENN’s embedding of target-ligand complex.

Original: The second action is to select the SMARTS reaction from the reaction set conditioned on the selectedligand in the first action.
Revised: The second action is to select the SMARTS reaction from the reaction set conditioned on the selected ligand in the first action.

Original: Specifically, for each reaction, we generate the new ligand X childmutation , thenobtain the embedding of target, first ligand X parentmutation and the new ligand X childmutation through ENN,concatenate these three embeddings and feed it into an MLP to estimate a scalar as unnormalized probability.
Revised: Specifically, for each reaction, we generate the new ligand X childmutation , then obtain the embedding of target, first ligand X parentmutation and the new ligand X childmutation through ENN, concatenate these three embeddings and feed it into an MLP to estimate a scalar as unnormalized probability.

Original: The unnormalized probabilities for all the reactions are normalized via softmax function, i.e., p (2) mutation ( ξ | X parentmutation , S ( t ) )
Revised: The unnormalized probabilities for all the reactions are normalized via softmax function, i.e., p (2) mutation ( ξ | X parentmutation , S ( t ) )

Original: =
Revised: =

Original: Softmax (cid:8) MLP ( h T ⊕ h X parentmutation ⊕ h X childmutation ]) , · · · , (cid:9) ξ ∈R , where
Revised: Softmax (cid:8) MLP ( h T ⊕ h X parentmutation ⊕ h X childmutation ]) , · · · , (cid:9) ξ ∈R , where

Original: X parentmutation mutated by ξ −−−−−−−→ X childmutation , R is the reaction set.
Revised: X parentmutation mutated by ξ −−−−−−−→ X childmutation , R is the reaction set.

Original: The probability of the generated ligand X childmutationis
Revised: The probability of the generated ligand X childmutation is p mutation ( X childmutation |S

Original: [Equation]
Revised: ( t ) )

Original: [Equation]
Revised: = p (1) mutation ( X parentmutation |S ( t ) ) · p (2) mutation ( ξ | X parentmutation , S ( t ) ) .

Original: Policy Gradient .
Revised: Policy Gradient .

Original: We leverage policy gradient to train the target-ligand policy neural network.
Revised: We leverage policy gradient to train the target-ligand policy neural network.

Original: Specifically, we consider maximizing the expected reward as objective via REINFORCE [28],
Revised: Specifically, we consider maximizing the expected reward as objective via REINFORCE [29],

Original: [Equation]
Revised: [Equation]

Original: where p ( X ) is defined in Equation (2) and (3) for crossover and mutation, respectively.
Revised: where p ( X ) is defined in Equation (2) and (3) for crossover and mutation, respectively.

Original: The wholepipeline is illustrated in Figure 1.
Revised: The whole pipeline is illustrated in Figure 1.

Original: To provide a warm start and leverage the structural information,we pretrain the ENNs on 3D target-ligand binding affinity prediction task, where the inputs are thetarget-ligand complexes, and the outputs are their binding affinity.
Revised: To provide a warm start and leverage the structural information, we pretrain the ENNs on 3D target-ligand binding affinity prediction task, where the inputs are the target-ligand complexes, and the outputs are their binding affinity.

Original:  Experiment
Revised:  Experiment

Original: In this section, we briefly describe the experimental setup and results.
Revised: In this section, we briefly describe the experimental setup and results.

Original: The Appendix includes moredetails, including software configuration, implementation details, dataset description & processing,hyperparameter tuning,code repository, ablation study, and additional experimental results. 
Revised: The Appendix includes more details, including software configuration, implementation details, dataset description & processing, hyperparameter tuning, ablation study, and additional experimental results. The code is available at https://github.com/futianfan/reinforced-genetic-algorithm .

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Docking Simulation .
Revised: Docking Simulation .

Original: We adopt AutoDock Vina [51] to evaluate the binding affinity.
Revised: We adopt AutoDock Vina [52] to evaluate the binding affinity.

Original: The dockingscore estimated by AutoDock Vina is called Vina score and roughly characterizes the free energychanges of binding processes in kcal/mol.
Revised: The docking score estimated by AutoDock Vina is called Vina score and roughly characterizes the free energy changes of binding processes in kcal/mol.

Original: Thus lower Vina score means a stronger binding affinitybetween the ligand and target.
Revised: Thus lower Vina score means a stronger binding affinity between the ligand and target.

Original: We picked various disease-related proteins, including G-proteincoupling receptors (GPCRs) and kinases from DUD-E [52] and the SARS-CoV-2 main protease [53]as targets.
Revised: We picked various disease-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E [53] and the SARS-CoV-2 main protease [54] as targets.

Original: Please see the Appendix for more information.
Revised: Please see the Appendix for more information.

Original: Baselines .
Revised: Baselines .

Original: The baseline methods cover traditional brute-force search methods (Screening), deepgenerative models (JTVAE and Gen3D), genetic algorithm (GA+D, graph-GA, Autogrow 4.0),reinforcement learning methods (MolDQN, RationaleRL, REINVENT, GEGL), and MCMC method(MARS).
Revised: The baseline methods cover traditional brute-force search methods (Screening), deep generative models (JTVAE and Gen3D), genetic algorithm (GA+D, graph-GA, Autogrow 4.0), reinforcement learning methods (MolDQN, RationaleRL, REINVENT, GEGL), and MCMC method (MARS).

Original: Gen3D and Autogrow 4.0 are structure-based drug design methods, while others aregeneral-purpose molecular design methods.
Revised: Gen3D and Autogrow 4.0 are structure-based drug design methods, while others are general-purpose molecular design methods.

Original: Although methods explicitly utilizing target structuresare relatively few, we add general-purpose molecular design methods optimizing the same dockingoracle scores as ours, which is a common use case, as baselines [15, 14].
Revised: Although methods explicitly utilizing target structures are relatively few, we add general-purpose molecular design methods optimizing the same docking oracle scores as ours, which is a common use case, as baselines [16, 14].

Original: Concretely, Screeningmimics high throughput screening via sampling from ZINC database randomly; JTVAE (Junction 
Revised: Concretely, Screening mimics high throughput screening via sampling from ZINC database randomly; JTVAE (Junction Tree Variational

Original: Tree Variational Auto-Encoder) [20] uses a Bayesian optimization on the latent space to indirectlyoptimize molecules;Gen3D [10] is an auto-regressive generative model that grows 3D structuresatom-wise inside the binding pocket; GA+D
Revised: Auto-Encoder) [21] uses a Bayesian optimization on the latent space to indirectly optimize molecules;

Original: [26] represents molecule as SELFIES string [54] anduses genetic algorithm enhanced by a discriminator neural network; Graph-GA [15] conduct geneticalgorithm on molecular graph representation; Autogrow 4.0 [16] is the state-of-the-art geneticalgorithm in structure-based drug design; MolDQN (Molecule Deep Q-Network) [30] leverages deep Q-value learning to grow molecules atom-wisely; RationaleRL [31] uses rationale (e.g., functionalgroups or subgraphs) as the building block and a policy gradient method to guide the training ofgraph neural network-based generator; REINVENT [28] represent molecules as SMILES stringand uses policy gradient based reinforcement learning methods to guide the training of the RNNgenerator; GEGL (genetic expert-guided learning)
Revised: Gen3D [10] is an auto-regressive generative model that grows 3D structures atom-wise inside the binding pocket; GA+D [27] represents molecule as SELFIES string [55] and uses genetic algorithm enhanced by a discriminator neural network; Graph-GA [16] conduct genetic algorithm on molecular graph representation; Autogrow 4.0 [17] is the state-of-the-art genetic algorithm in structure-based drug design; MolDQN (Molecule Deep Q-Network) [31] leverages deep Q-value learning to grow molecules atom-wisely; RationaleRL [32] uses rationale (e.g., functional groups or subgraphs) as the building block and a policy gradient method to guide the training of graph neural network-based generator; REINVENT [29] represent molecules as SMILES string and uses policy gradient based reinforcement learning methods to guide the training of the RNN generator; GEGL (genetic expertguided learning)

Original: [32] uses LSTM guided by reinforcement learningto imitate the GA exploration; MARS (Markov Molecule Sampling)
Revised: [33] uses LSTM guided by reinforcement learning to imitate the GA exploration; MARS (Markov Molecule Sampling)

Original: [35] leverages Markov chain Monte Carlo sampling (MCMC) with adaptive proposal and annealing scheme to search chemicalspace.
Revised: [36] leverages Markov chain Monte Carlo sampling (MCMC) with adaptive proposal and annealing scheme to search chemical space.

Original: To conduct a fair comparison, we limit the number of oracle calls to 1,000 times for eachmethod.
Revised: To conduct a fair comparison, we limit the number of oracle calls to 1,000 times for each method.

Original: Implementation details of all the baseline methods are available in Appendix.
Revised: All the baselines can be run with one-line code using the software ( https://github.com/wenhao-gao/mol_opt ) in practical molecular optimization benchmark [15].

Original: Dataset : we randomly select molecules from ZINC [55] database (around 250 thousands drug-likemolecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, GA+D). 
Revised: Dataset : we randomly select molecules from ZINC [56] database (around 250 thousands drug-like molecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, GA+D). ZINC also [Table]

Original: ZINC alsoserves as the training data for pretraining the model in JTVAE, REINVENT, RationaleRL, etc.
Revised: serves as the training data for pretraining the model in JTVAE, REINVENT, RationaleRL, etc.

Original: Weadopt CrossDocked2020 [56] dataset that contains around 22 million ligand-protein complexes as[Table] the training data for pretraining the policy neural networks, as mentioned in Section 3.3.
Revised: We adopt CrossDocked2020 [57] dataset that contains around 22 million ligand-protein complexes as the training data for pretraining the policy neural networks, as mentioned in Section 3.3.

Original: Moredescriptions are available in Appendix.
Revised: More descriptions are available in Appendix.

Original: Metrics .
Revised: Metrics .

Original: The selection of evaluation metrics follows recent works in molecule optimization [20, 26,31, 35] and structure-based drug design [16, 10, 14].
Revised: The selection of evaluation metrics follows recent works in molecule optimization [21, 27, 32, 36] and structure-based drug design [17, 10, 14].

Original: For each method, we select top-100 moleculeswith the best docking scores for evaluation and consider the following metrics: TOP-1/10/100(average docking score of top-1/10/100 molecules): docking score directly measures the bindingaffinity between the ligand and target and is the most informative metric in structure-based drugdesign; Novelty (Nov) (% of the generated molecules that are not in training set); Diversity (Div)(average pairwise Tanimoto distance between the Morgan fingerprints); We also evaluate some simplepharmaceutical properties, including quantitative drug-likeness ( QED ) and synthetic accessibility( SA ).
Revised: For each method, we select top-100 molecules with the best docking scores for evaluation and consider the following metrics: TOP-1/10/ (average docking score of top-1/10/100 molecules): docking score directly measures the binding affinity between the ligand and target and is the most informative metric in structure-based drug design; Novelty (Nov) (% of the generated molecules that are not in training set); Diversity (Div) (average pairwise Tanimoto distance between the Morgan fingerprints); We also evaluate some simple pharmaceutical properties, including quantitative drug-likeness ( QED ) and synthetic accessibility ( SA ).

Original: QED score indicates drug-likeliness ranging from 0 to 1 (higher the better).
Revised: QED score indicates drug-likeliness ranging from 0 to 1 (higher the better).

Original: SA score rangesfrom 1 to 10 (lower the better).
Revised: SA score ranges from 1 to 10 (lower the better).

Revised: All the evaluation functions are available at Therapeutics data commons (TDC, https://tdcommons.ai/fct_overview )

Original: 4.2 Results
Revised: 4.2 Results

Original: Stronger Optimization Performance .
Revised: Stronger Optimization Performance .

Original: We summarized the main results of the structure-baseddrug design in Table 1.
Revised: We summarized the main results of the structure-based drug design in Table 1.

Original: We evaluate all the methods on all targets and report each metric’s meanand standard deviations across all targets.
Revised: We evaluate all the methods on all targets and report each metric’s mean and standard deviations across all targets.

Original: Our result shows RGA achieves the best performance in
Revised: Our result shows RGA achieves the best performance in

Original: TOP-100/10/1 scores among all methods we compared.
Revised: TOP-100/10/1 scores among all methods we compared.

Original: Compared to Autogrow 4.0, RGA ’s betterperformance in docking score demonstrates that the policy networks contribute positively to thechemical space navigation and eventually help discover more potent binding molecules.
Revised: Compared to Autogrow 4.0, RGA ’s better performance in docking score demonstrates that the policy networks contribute positively to the chemical space navigation and eventually help discover more potent binding molecules.

Original: On the otherhand, including longer-range navigation steps enabled by crossover leads to superior performancethan other RL methods (REINVENT, MolDQN, GEGL and RationaleRL) that only focus on localmodifications.
Revised: On the other hand, including longer-range navigation steps enabled by crossover leads to superior performance than other RL methods (REINVENT, MolDQN, GEGL and RationaleRL) that only focus on local modifications.

Original: In addition, we also observed competitive structure quality measured by QED ( > 0 . 7 )and SA_Score ( < 2 . 5 ) in Autogrow 4.0 and RGA without involving them as optimization objectives,thanks to the mutation steps originating from chemical reactions.
Revised: In addition, we also observed competitive structure quality measured by QED ( > 0 . 7 ) and SA_Score ( < 2 . 5 ) in Autogrow 4.0 and RGA without involving them as optimization objectives, thanks to the mutation steps originating from chemical reactions.

Original: We visualize two designed ligandswith optimal affinity for closer inspection in Figure 2(a) and 2(b), and find both ligands bind tightlywith the targets.
Revised: We visualize two designed ligands with optimal affinity for closer inspection in Figure 2(a) and 2(b), and find both ligands bind tightly with the targets.

Original: Suppressed Random-Walk Behavior .
Revised: Suppressed Random-Walk Behavior .

Original: Especially in SBDD, when the oracle functions are expensive molecular simulations, robustness to random seeds is essential for improving the worst-caseperformance of algorithms.
Revised: Especially in SBDD, when the oracle functions are expensive molecular simulations, robustness to random seeds is essential for improving the worst-case performance of algorithms.

Original: One of the major issues in traditional GAs is that they have a significant variance between multiple independent runs as they randomly select parents for crossover andmutation types.
Revised: One of the major issues in traditional GAs is that they have a significant variance between multiple independent runs as they randomly select parents for crossover and mutation types.

Original: To examine this behavior, we run five independent runs for RGA , Autogrow 4.0 andgraph-GA (three best baselines, all are GA methods) on all targets and plot the standard deviationsbetween runs in Figure 2(c) and 2(d).
Revised: To examine this behavior, we run five independent runs for RGA , Autogrow 4.0 and graph-GA (three best baselines, all are GA methods) on all targets and plot the standard deviations between runs in Figure 2(c) and 2(d).

Original: With policy networks guiding the action steps, we observed thatthe random-walk behavior in Autogrow 4.0 was suppressed in RGA , indicated by the smaller variance.
Revised: With policy networks guiding the action steps, we observed that the random-walk behavior in Autogrow 4.0 was suppressed in RGA , indicated by the smaller variance.

Original: Especially in the later learning phase (after 500 oracle calls), the policy networks are fine-tuned andguide the search more intelligently.
Revised: Especially in the later learning phase (after 500 oracle calls), the policy networks are fine-tuned and [Figure] guide the search more intelligently.

Original: This advantage leads to improved worst-case performance and ahigher probability of successfully identifying bioactive drug candidates with constrained resources.[Figure]
Revised: This advantage leads to improved worst-case performance and a higher probability of successfully identifying bioactive drug candidates with constrained resources.

Original: Knowledge Transfer Between Protein Targets .
Revised: Knowledge Transfer Between Protein Targets .

Original: To verify if RGA benefited from learning the sharedphysics of ligand-target interaction, we conducted an ablation study whose results are in the lasttwo rows of Table 1.
Revised: To verify if RGA benefited from learning the shared physics of ligand-target interaction, we conducted an ablation study whose results are in the last two rows of Table 1.

Original: Specifically, we compare RGA with two variants: (1) RGA -pretrain that does notpretrain the policy network with all native complex structures in the CrossDocked2020; (2) RGA -KT(knowledge transfer) that fine-tune the networks with data of individual target independently.
Revised: Specifically, we compare RGA with two variants: (1) RGA -pretrain that does not pretrain the policy network with all native complex structures in the CrossDocked2020; (2) RGA -KT (knowledge transfer) that fine-tune the networks with data of individual target independently.

Original: Wefind that both strategies positively contribute to RGA on TOP-100/10/1 docking score.
Revised: We find that both strategies positively contribute to RGA on TOP-100/10/1 docking score.

Original: These resultsdemonstrate the policy networks successfully learn the shared physics of ligand-target interactionsand leverage the knowledge to improve their performance.
Revised: These results demonstrate the policy networks successfully learn the shared physics of ligand-target interactions and leverage the knowledge to improve their performance.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose Reinforced Genetic Algorithm ( RGA ) to tackle the structure-based drugdesign problem.
Revised: In this paper, we propose Reinforced Genetic Algorithm ( RGA ) to tackle the structure-based drug design problem.

Original: RGA reformulate the evolutionary process in genetic algorithms as a Markov decisionprocess called evolutionary Markov decision process (EMDP) so that the searching processes couldbenefit from trained neural models.
Revised: RGA reformulate the evolutionary process in genetic algorithms as a Markov decision process called evolutionary Markov decision process (EMDP) so that the searching processes could benefit from trained neural models.

Original: Specifically, we train policy networks to choose the parents tocrossover and mutate instead of randomly sampling them.
Revised: Specifically, we train policy networks to choose the parents to crossover and mutate instead of randomly sampling them.

Original: Further, we also leverage the commonphysics of the ligand-target interaction and adopt a knowledge-transfer strategy that uses data fromother targets to train the networks.
Revised: Further, we also leverage the common physics of the ligand-target interaction and adopt a knowledge-transfer strategy that uses data from other targets to train the networks.

Original: Through empirical study, we show that RGA has strong and robustoptimization performance, consistently outperforming baseline methods in terms of docking score.
Revised: Through empirical study, we show that RGA has strong and robust optimization performance, consistently outperforming baseline methods in terms of docking score.

Original: Though we adopted mutations originating from chemical reactions and the structural quality metricsseem good, we need to emphasize that the designed molecules from RGA do not guarantee synthesizability [48], as the crossover operations may break inheriting synthesizability.
Revised: Though we adopted mutations originating from chemical reactions and the structural quality metrics seem good, we need to emphasize that the designed molecules from RGA do not guarantee synthesizability [49], as the crossover operations may break inheriting synthesizability.

Original: Directly working onsynthetic pathways could solve the problem [27, 57], but the extension is not trivial.
Revised: Directly working on synthetic pathways could solve the problem [28, 59], but the extension is not trivial.

Original: As for futuredirection, we expect to theoretically analyze the EMDP formulation and the performance of RGA .
Revised: As for future direction, we expect to theoretically analyze the EMDP formulation and the performance of RGA .

Revised: Acknowledgments and Disclosure of Funding

Original: Wealso expect to generalize RGA to other combinatorial optimization scenarios, such as symbolic lawsdiscovery [58], quantum circuits design [59]), etc.
Revised: T.F. and J.S. were supported by NSF award SCH-2205289, SCH-2014438, IIS-1838042, NIH award R01 1R01NS107291-01. W.G. and C.C. were supported by the Office of Naval Research under grant number N00014-21-1-2195 and the Machine Learning for Pharmaceutical Discovery and Synthesis consortium. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Office of Naval Research.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: Contents 1 Introduction 2 Related Works 3 Method 33.
Revised: Contents 1 Introduction 2 Related Works 3 Method 3.

Original: Evolutionary Process . .
Revised: Evolutionary Process . .

Original: Evolutionary Markov Decision Process . .
Revised: Evolutionary Markov Decision Process . .

Original: Target-Ligand Policy Network . .
Revised: Target-Ligand Policy Network . .

Original: 54 Experiment 74.
Revised: 4 Experiment 4.

Original: Experimental Setup . .
Revised: Experimental Setup . .

Original: Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 85 Conclusion[-
-]
Revised: 5 Conclusion[-
-]

Original: A Mathematical Notation[-
-]
Revised: A Mathematical Notation[-
-]

Original: B Illustration of Genetic Algorithm[-
-]
Revised: B Illustration of Genetic Algorithm[-
-]

Original: C Baseline Setup[-
-]
Revised: C Baseline Setup[-
-]

Original: D Additional Experimental Setup[-
-]
Revised: D Additional Experimental Setup[-
-]

Original: D.1 Docking Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1 Docking Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 Dataset . .
Revised: D.2 Dataset . .

Original: D.3 Evaluation metrics . .
Revised: D.3 Evaluation metrics . .

Original: E Implementation Details[-
-]
Revised: E Implementation Details[-
-]

Original: E.1 Software/Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.1 Software/Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2 Hyperparameter Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.2 Hyperparameter Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.3 Code Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.3 Code Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Additional Experiment
Revised: F Additional Experiment

Original: F.
Revised: F.

Original: Efficiency Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Efficiency Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Pretraining Equivariant neural network . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Pretraining Equivariant neural network . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Additional Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Additional Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Scalability . . . .
Revised: Scalability . . . .

Original: F.
Revised: F.

Original: Significance Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Significance Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Example of the generated ligand . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Example of the generated ligand . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G Additional Discussion on Related Work
Revised: G Additional Discussion on Related Work

Original: A Mathematical Notation
Revised: A Mathematical Notation

Original: For ease of exposition, we list the mathematical notations in Table 2.
Revised: For ease of exposition, we list the mathematical notations in Table 2.

Original: All the mathematical notations are divided into three parts: (1) notation for genetic algorithm (Section 3.1); (2) notation forequivariance neural networks (ENN) [18] (Section 3.3); (3) notations for policy network (Section 3.3).
Revised: All the mathematical notations are divided into three parts: (1) notation for genetic algorithm (Section 3.1); (2) notation for equivariance neural networks (ENN) [19] (Section 3.3); (3) notations for policy network (Section 3.3).

Original: [Table]
Revised: [Table]

Revised: ⊕

Revised: Z

Original: that embeds the i -th node
Revised: [Equation] H [Equation] Node

Original: Z ( l ) =
Revised: [Table] at

Original: { z ( l ) i } Ni =
Revised: [Table] [Table]

Original: Coordinate embeddings at the l -th layer w ( l ) ij message vector for the edge from node  ito node j at l -th layer v ( l ) i message vector
Revised: h [Equation] The initial node embedding that embeds the i -th node Z

Original: [Table]
Revised: [Equation]

Original: node i
Revised: l [Equation] l [Equation]

Revised: i

Original: [Table]
Revised: [Equation]

Original: th layer z ( l ) i the position embedding for node i at  l -th layer h ( l ) i the node embedding for  node i  at l -th layerh
Revised: Coordinate embeddings at the l -th layer w [Equation] message vector for the edge from node i to node j at l -th layer

Original: Y = ENN ( Y )
Revised: [Equation]

Original: ENN representation of the 3D graph Y (Equation 1) p (1)
Revised: message vector for node i at l -th layer z [Equation] i the position embedding for node i at l -th layer h ( l

Original: [Table]
Revised: [Equation]

Original: ( X parent 1 crossover |S ( t ) ) [Table] crossover p
Revised: i the node embedding for node i at l -th layer h Y = ENN

Original: [Table] [Table]
Revised: [Equation]

Original: | X
Revised: Y )

Original: [Table] crossover , S
Revised: ENN representation of the 3D graph Y (Equation 1)

Original: [Table]
Revised: [Equation]

Original: t ) ) 
Revised: ) probability to select the first parent molecule in crossover p

Original: [Table]
Revised: [Equation]

Original: p

Original: [Table]
Revised: [Equation]

Original: S

Original: [Table]
Revised: [Equation]

Original: S

Original: [Table]
Revised: [Equation]

Original: p [Table]
Revised: t ) )

Original: [Table] t [Table] [Table]
Revised: probability to select the second parent molecule in crossover p crossover ( X child 1 crossover , X child 2 crossover |S ( t ) ) probability of two generated child molecules in crossover (Eq 2) p (1) mutation ( X parentmutation |S ( t ) ) probability to select the parent molecule in mutation p (2) mutation ( ξ | X parentmutation , S ( t ) ) probability to select the reaction in mutation p mutation ( X childmutation |S ( t ) ) probability of generated child molecule in mutation (Eq 3)

Original: B Illustration of Genetic Algorithm
Revised: B Illustration of Genetic Algorithm

Original: Figure 3 provides two examples to illustrate crossover and mutation operations in genetic algorithmdescribed in Section 3.1.
Revised: Figure 3 provides two examples to illustrate crossover and mutation operations in genetic algorithm described in Section 3.1.

Original: Crossover , also called recombination, combines the structure of two parents to generate new children.
Revised: Crossover , also called recombination, combines the structure of two parents to generate new children.

Original: Following Autogrow 4.0 [16], as shown in Figure 3(a), we select two parents from the last generationand search for the largest common substructure shared between them.
Revised: Following Autogrow 4.0 [17], as shown in Figure 3(a), we select two parents from the last generation and search for the largest common substructure shared between them.

Original: Then we generate two childrenby randomly switching their decorating moieties, i.e., the side chains attached to the commonsubstructure.
Revised: Then we generate two children by randomly switching their decorating moieties, i.e., the side chains attached to the common substructure.

Original: Mutation operator performs an in silico chemical reaction to generate an altered child compound(i.e. , product in the chemical reaction) derived from a parent (reactant in the chemical reaction), asshown in Figure 3(b).
Revised: Mutation operator performs an in silico chemical reaction to generate an altered child compound (i.e., product in the chemical reaction) derived from a parent (reactant in the chemical reaction), as shown in Figure 3(b).

Original: The chemical reaction here contains two reactants, one is parent molecule,another is from reaction set.
Revised: The chemical reaction here contains two reactants, one is parent molecule, another is from reaction set.

Original: The reaction set R is generated via merging two public reaction libraries:(1) the AutoClickChemRxn set (36 reactions) [46] and (2) RobustRxn set (58 reactions [49]).
Revised: The reaction set R is generated via merging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions) [47] and (2) RobustRxn set (58 reactions [50]).

Original: Eachreaction ξ in reaction set R contains a SMARTS string based reaction template and a reactant.
Revised: Each reaction ξ in reaction set R contains a SMARTS string based reaction template and a reactant.

Original: It uses
Revised: It uses

Original: SMARTS reaction template, together with RDKit [60], to perform chemical mutations efficiently.
Revised: SMARTS reaction template, together with RDKit [60], to perform chemical mutations efficiently.

Original: The process is written as X parentmutation mutated by ξ −−−−−−−→ X childmutation , where ξ is selected reaction (with a reactiontemplate and another reactant).
Revised: The process is written as X parentmutation mutated by ξ −−−−−−−→ X childmutation , where ξ is selected reaction (with a reaction template and another reactant).

Original: The ligand to be mutated and the reaction used for mutation are bothrandomly selected from previous generation and reaction set R , respectively.
Revised: The ligand to be mutated and the reaction used for mutation are both randomly selected from previous generation and reaction set R , respectively.

Original: Compared with themutation operator in conventional GA that randomly flipping an arbitrary bit, the reaction-basedmutation enhance synthesizability of the generated molecules [16].
Revised: Compared with the mutation operator in conventional GA that randomly flipping an arbitrary bit, the reaction-based mutation enhance synthesizability of the generated molecules [17].

Original: Mutation operator performs an insilico chemical reaction to generate an altered child compound (i.e., product in the chemical reaction)derived from a parent (reactant in the chemical reaction).
Revised: Mutation operator performs an in silico chemical reaction to generate an altered child compound (i.e., product in the chemical reaction) derived from a parent (reactant in the chemical reaction).

Original: The chemical reaction here contains tworeactants, one is parent molecule, another is from reaction set.
Revised: The chemical reaction here contains two reactants, one is parent molecule, another is from reaction set.

Original: [Figure]
Revised: [Figure]

Revised: -reaction template (with another reactant) to performs an in silico chemical reaction to generate child compound (i e., product).

Original: C Baseline Setup
Revised: C Baseline Setup

Original: In this section, we describe the experimental setting for baseline methods.
Revised: In this section, we describe the experimental setting for baseline methods.

Original: Most of the settings followthe original papers.
Revised: Most of the settings follow the original papers.

Original: • GA+D
Revised: • GA+D

Original: (genetic algorithm enhanced by discriminator neural network) [26] utilizes SELFIESstring as the representation of molecules, thus guaranteeing the 100% chemical validity ofthe generated molecules.
Revised: (genetic algorithm enhanced by discriminator neural network) [27] utilizes SELFIES string as the representation of molecules, thus guaranteeing the 100% chemical validity of the generated molecules.

Original: Following their original paper, the discriminator neural network isa two-layer fully connected neural network with ReLU activation and sigmoid output layer.
Revised: Following their original paper, the discriminator neural network is a two-layer fully connected neural network with ReLU activation and sigmoid output layer.

Original: The hidden size is 100.
Revised: The hidden size is 100.

Original: the size of the output layer is 1.
Revised: the size of the output layer is 1.

Original: The input feature of discriminator neural network is a vector of chemical and geometrical properties characterizing themolecules.
Revised: The input feature of discriminator neural network is a vector of chemical and geometrical properties characterizing the molecules.

Original: The population size is set to 300.
Revised: The population size is set to 300.

Original: Maximal generation number is set to 1000.The patience is set to 5.
Revised: Maximal generation number is set to 1000. The patience is set to 5.

Original: When the property does not improve when the patience exhausts,the process early stops.
Revised: When the property does not improve when the patience exhausts, the process early stops.

Original: We used Adam optimizer with 1e-3 as the initial learning rate.
Revised: We used Adam optimizer with 1e-3 as the initial learning rate.

Original: beta( β ) is the weight of discriminator neural network’s score in fitness evaluation, which is usedto select most promising molecules in each generation.
Revised: beta ( β ) is the weight of discriminator neural network’s score in fitness evaluation, which is used to select most promising molecules in each generation.

Original: We set β = 1 .•
Revised: We set β = 1 .

Original:  Graph-GA [15] uses molecular graph to represent molecules and uses crossover andmutation operations to edit the molecular graph.
Revised: • Graph-GA [16] uses molecular graph to represent molecules and uses crossover and mutation operations to edit the molecular graph.

Original: After tuning, the size of population is setto 120.
Revised: After tuning, the size of population is set to 120.

Original: The size of offspring is set to 70.
Revised: The size of offspring is set to 70.

Original: The mutation rate is set to 0.067.
Revised: The mutation rate is set to 0.067.

Original: Graph-GAdo not have learnable parameters and is easy to implement.
Revised: Graph-GA do not have learnable parameters and is easy to implement.

Original: We use the implementation inGuacaMol [13].• MolDQN (Molecule Deep Q-Networks)
Revised: We use the implementation in GuacaMol [13].

Original: [30] uses molecular graph to represent molecules,formulate the molecule optimization process as a Markov Decision Process (MDP) and utilize Deep Q-value learning to optimize it.
Revised: • MolDQN (Molecule Deep Q-Networks) [31] uses molecular graph to represent molecules, formulate the molecule optimization process as a Markov Decision Process (MDP) and utilize Deep Q-value learning to optimize it.

Original: It grows molecular graph atom-wise, that is,[Table] in each episode, it adds one atom to the partially generated molecular graph.
Revised: It grows molecular graph atom-wise, that is, in each episode, it adds one atom to the partially generated molecular graph.

Original: The reward[Table] function is the negative value of the Vina score.
Revised: The reward function is the negative value of the Vina score.

Original: Following the original paper, maximalstep in each episode is 40.
Revised: Following the original paper, maximal step in each episode is 40.

Original: Each step calls oracle once.
Revised: Each step calls oracle once.

Original: The discount factor is 0.9.
Revised: The discount factor is 0.9.

Original: DeepQ-network is a multilayer perceptron (MLP) whose hidden dimensions are 1024, 512, 128,32, respectively.
Revised: Deep Q-network is a multilayer perceptron (MLP) whose hidden dimensions are 1024, 512, 128, 32, respectively.

Original: The model size is 6.4 M. The input of the Q-network is the concatenationof the molecule feature (2048-bit Morgan fingerprint, with a radius of 3) and the number ofleft steps.
Revised: The model size is 6.4 M. The input of the Q-network is the concatenation of the molecule feature (2048-bit Morgan fingerprint, with a radius of 3) and the number of left steps.

Original: Adam is used as an optimizer with 1e-4 as the initial learning rate.
Revised: Adam is used as an optimizer with 1e-4 as the initial learning rate.

Original: Only ringswith a size of 5 and 6 are allowed.
Revised: Only rings with a size of 5 and 6 are allowed.

Original: It leverages ϵ -greedy together with randomized value[Table] functions (bootstrapped-DQN) as an exploration policy, ϵ is annealed from 1 to 0.01 in apiecewise linear way.• RationaleRL 
Revised: It leverages ϵ -greedy together with randomized value functions (bootstrapped-DQN) as an exploration policy, ϵ is annealed from 1 to 0.01 in a piecewise linear way. • RationaleRL [32].

Original: The architecture of the generator is a message-passing network (MPN)followed by MLPs applied in breadth-first order.
Revised: The architecture of the generator is a message-passing network (MPN) followed by MLPs applied in breadth-first order.

Original: The generator is pre-trained on generalmolecules combined with an encoder and then fine-tuned to maximize the reward functionusing policy gradient.
Revised: The generator is pre-trained on general molecules combined with an encoder and then fine-tuned to maximize the reward function using policy gradient.

Original: The encoder and decoder MPNs both have hidden dimensions of 400.The dimension of the latent variable is 50.
Revised: The encoder and decoder MPNs both have hidden dimensions of 400. The dimension of the latent variable is 50.

Original: Adam optimizer is used on both pre-training andfine-tuning with initial learning rates of 1e-3, 5e-4, respectively.
Revised: Adam optimizer is used on both pre-training and fine-tuning with initial learning rates of 1e-3, 5e-4, respectively.

Original: The annealing rate is 0.9.We pre-trained the model with 20 epochs.
Revised: The annealing rate is 0.9. We pre-trained the model with 20 epochs.

Original: [Table] • MARS [35] leverage Markov chain Monte Carlo sampling (MCMC) on molecules with[Table] an annealing scheme and an adaptive proposal.
Revised: • MARS [36] leverage Markov chain Monte Carlo sampling (MCMC) on molecules with an annealing scheme and an adaptive proposal.

Original: The proposal is parameterized by a graph[Table] neural network, which is trained on MCMC samples.
Revised: The proposal is parameterized by a graph neural network, which is trained on MCMC samples.

Original: We follow most of the settings in the[Table] original paper.
Revised: We follow most of the settings in the original paper.

Original: The message passing network has six layers, where the node embedding[Table] size is set to 64.
Revised: The message passing network has six layers, where the node embedding size is set to 64.

Original: Adam is used as an optimizer with 3e-4 initial learning rate.
Revised: Adam is used as an optimizer with 3e-4 initial learning rate.

Original: To generate abasic unit, top-1000 frequent fragments are drawn from ZINC database [55] by enumeratingsingle bonds to break.
Revised: To generate a basic unit, top-1000 frequent fragments are drawn from ZINC database [56] by enumerating single bonds to break.

Original: During the annealing process, the temperature T = 0 .
Revised: During the annealing process, the temperature T = 0 .

Original: ⌊ t/ 5 ⌋ wouldgradually decrease to 0.•
Revised: ⌊ t/ 5 ⌋ would gradually decrease to 0.

Original:  Autogrow 4.0 [16] is the base model for RGA and have been briefly described in Section 3.1.The setup of Autogrow is the same as RGA for fair comparison, the only difference is thatRGA use policy network to guide the selection of ligands and reaction for crossover andmutation while Autogrow randomly selects them.
Revised: • Autogrow 4.0 [17] is the base model for RGA and have been briefly described in Section 3.1. The setup of Autogrow is the same as RGA for fair comparison, the only difference is that RGA use policy network to guide the selection of ligands and reaction for crossover and mutation while Autogrow randomly selects them.

Original: The reaction set R is generated viamerging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions) [46]and (2) RobustRxn set (58 reactions [49]).
Revised: The reaction set R is generated via merging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions) [47] and (2) RobustRxn set (58 reactions [50]).

Original: In each generation, It generates 200 offspring(100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vinascores) ones for the next generation.•
Revised: In each generation, It generates 200 offspring (100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vina scores) ones for the next generation.

Original:  Screening exhaustively searches the ZINC database [55] within oracle budget.
Revised: • Screening exhaustively searches the ZINC database [56] within oracle budget.

Original: It is thetraditional high-throughput screening approach.• REINVENT [28] is a reinforcement learning approach, represent molecule as SMILESstring and uses recurrent neural network to model SMILES string.
Revised: It is the traditional high-throughput screening approach. • REINVENT [29] is a reinforcement learning approach, represent molecule as SMILES string and uses recurrent neural network to model SMILES string.

Original: It pretrains a priormodel using molecules on ZINC and finetune the model using the reward function.
Revised: It pretrains a prior model using molecules on ZINC and finetune the model using the reward function.

Original: It usesREINFORCE to maximize the expected reward function.
Revised: It uses REINFORCE to maximize the expected reward function.

Original: The learning rate is set to 0.0005;the batch size is set to 64; The hyperparameter σ weighs the pretrained prior model and thereward function, and is set to 60.
Revised: The learning rate is set to 0.0005; the batch size is set to 64; The hyperparameter σ weighs the pretrained prior model and the reward function, and is set to 60.

Original: The model size is 16.3M.• JTVAE [20] build a junction tree to represent molecule via using substructure (either ringor atom) to represent molecule.
Revised: The model size is 16.3M. • JTVAE [21] build a junction tree to represent molecule via using substructure (either ring or atom) to represent molecule.

Original: It uses both molecular graph-leven and junction tree-levelencoder and decoder.
Revised: It uses both molecular graph-leven and junction tree-level encoder and decoder.

Original: The VAE model is pretrained on ZINC databases.
Revised: The VAE model is pretrained on ZINC databases.

Original: Then BayesianOptimization is used to optimize the docking score on the continuous latent space.
Revised: Then Bayesian Optimization is used to optimize the docking score on the continuous latent space.

Original: Weuse “botorch”, the python’s Bayesian optimization package, to implement the Bayesianoptimization process.
Revised: We use “botorch”, the python’s Bayesian optimization package, to implement the Bayesian optimization process.

Original: It has 703 substructures in vocabulary, extracted from ZINC.
Revised: It has 703 substructures in vocabulary, extracted from ZINC.

Original: Thehidden size is 450.
Revised: The hidden size is 450.

Original: The latent size of VAE is set to 56.
Revised: The latent size of VAE is set to 56.

Original: The model size is 21.8 M.• Gen3D [10] uses 3D deep generative models and grow the molecule via adding atomsauto-regressively.
Revised: The model size is 21.8 M. • Gen3D [10] uses 3D deep generative models and grow the molecule via adding atoms auto-regressively.

Original: It train a universal model for all the targets.
Revised: It train a universal model for all the targets.

Original: The number of messagepassing layers in context encoder is 6, and the hidden dimension is 256.
Revised: The number of message passing layers in context encoder is 6, and the hidden dimension is 256.

Original: We train the modelusing the Adam optimizer at learning rate 0.0001.
Revised: We train the model using the Adam optimizer at learning rate 0.0001.

Original: The model size is 17.4 M.• GEGL (Genetic Expert Guided Learning)
Revised: The model size is 17.4 M. • GEGL (Genetic Expert Guided Learning)

Original: [32] uses LSTM (guided by RL agent) to imitateGA process, however, it is unable to inherit the GA’s flexible assembling manner due to the auto-regressive essence of LSTM.
Revised: [33] uses LSTM (guided by RL agent) to imitate GA process, however, it is unable to inherit the GA’s flexible assembling manner due to the auto-regressive essence of LSTM.

Original: It use Adam as optimizer with initial learning rate 1e-3.
Revised: It use Adam as optimizer with initial learning rate 1e-3.

Original: The batch size during sampling is 512, the batch size during optimization is 256.
Revised: The batch size during sampling is 512, the batch size during optimization is 256.

Original: In GA,mutation rate is 0.01.
Revised: In GA, mutation rate is 0.01.

Original: The similarity threshold is 0.4, which constrain the similarity betweenthe original molecule and the edited molecule.
Revised: The similarity threshold is 0.4, which constrain the similarity between the original molecule and the edited molecule.

Original: The maximal SMILES length is set to 120.
Revised: The maximal SMILES length is set to 120.

Original: D Additional Experimental SetupD.1 Docking Simulation
Revised: D Additional Experimental Setup D.1 Docking Simulation

Original: Molecular docking is a computational method which predicts the preferred orientation of one moleculeto a second when a ligand and a target are bound to each other to form a stable complex.
Revised: Molecular docking is a computational method which predicts the preferred orientation of one molecule to a second when a ligand and a target are bound to each other to form a stable complex.

Original: Knowledgeof the preferred orientation in turn may be used to predict the strength of association or bindingaffinity between two molecules using, for example, scoring functions.
Revised: Knowledge of the preferred orientation in turn may be used to predict the strength of association or binding affinity between two molecules using, for example, scoring functions.

Original: We adopt AutoDock Vina [51]to evaluate the binding affinity.
Revised: We adopt AutoDock Vina [52] to evaluate the binding affinity.

Original: The docking score estimated by AutoDock Vina is called Vina scoreand roughly characterizes the free energy changes of binding processes in kcal/mol.
Revised: The docking score estimated by AutoDock Vina is called Vina score and roughly characterizes the free energy changes of binding processes in kcal/mol.

Original: Vina score isusually smaller than 0 and lower Vina score means a stronger binding affinity between the ligand andtarget.
Revised: Vina score is usually smaller than 0 and lower Vina score means a stronger binding affinity between the ligand and target.

Original: We leverage the negative value of the docking score as reward function (Equation 4).
Revised: We leverage the negative value of the docking score as reward function (Equation 4).

Original: D.
Revised: D.2 Dataset

Original: 2 Dataset

Original: In this paper, we use ZINC [55] database and CrossDocked2020 [56] dataset.
Revised: In this paper, we use ZINC [56] database and CrossDocked2020 [57] dataset.

Original: ZINC is a free databaseof 250 thousands commercially-available drug-like chemical compounds for virtual screening [55].
Revised: ZINC is a free database of 250 thousands commercially-available drug-like chemical compounds for virtual screening [56].

Original: We randomly select molecules from ZINC [55] database (around 250 thousands drug-like molecules)as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, graph-GA GA+D).
Revised: We randomly select molecules from ZINC [56] database (around 250 thousands drug-like molecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, graph-GA GA+D).

Original: Other baselinemethods also use ZINC to either pretrain the models, e.g., JTVAE, REINVENT, RationaleRL orprovide searching database, e.g., screening.
Revised: Other baseline methods also use ZINC to either pretrain the models, e.g., JTVAE, REINVENT, RationaleRL or provide searching database, e.g., screening.

Original: We adopt CrossDocked2020 [56] dataset that containsaround 22 million ligand-protein complexes as the training data for pretraining the policy neuralnetworks, as mentioned in Section 3.3.
Revised: We adopt CrossDocked2020 [57] dataset that contains around 22 million ligand-protein complexes as the training data for pretraining the policy neural networks, as mentioned in Section 3.3.

Original: Regarding the target proteins, we picked various disease-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E [52] and the SARS-CoV-2 main protease [53] astargets.
Revised: Regarding the target proteins, we picked various disease-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E [53] and the SARS-CoV-2 main protease [54] as targets.

Original: for all the selected target protein, the binding pocket size for all the targets are set to (15.0,15.0, 15.0).
Revised: for all the selected target protein, the binding pocket size for all the targets are set to (15.0, 15.0, 15.0).

Original: The units of coordinate are Angstrom ˚ A ( 10 − 10 m).
Revised: The units of coordinate are Angstrom ˚ A ( 10 − 10 m).

Original: Detailed descriptions of these targetsare available at https://www.rcsb.org/ .
Revised: Detailed descriptions of these targets are available at https://www.rcsb.org/ .

Original: D.3 Evaluation metrics
Revised: D.3 Evaluation metrics

Original: We leverage the following evaluation metrics to measure the optimization performance:
Revised: We leverage the following evaluation metrics to measure the optimization performance:

Original: • Novelty is the fraction of the generated molecules that do not appear in the training set.
Revised: • Novelty is the fraction of the generated molecules that do not appear in the training set.

Original: • Diversity of generated molecules is defined as the average pairwise Tanimoto distance between the Morgan fingerprints [29, 31, 35].
Revised: • Diversity of generated molecules is defined as the average pairwise Tanimoto distance between the Morgan fingerprints [30, 32, 36].

Original: [Equation]
Revised: [Equation]

Original: where M is the set of generated molecules that we want to evaluate.
Revised: where M is the set of generated molecules that we want to evaluate.

Original: sim ( m 1 , m 2 ) is the Tanimoto similarity between molecule m 1 and m 2 , where (Tanimoto) Similarity mea- sures the similarity between the input molecule and generated molecules.
Revised: sim ( m 1 , m 2 ) is the Tanimoto similarity between molecule m 1 and m 2 , where (Tanimoto) Similarity measures the similarity between the input molecule and generated molecules.

Original: It is defined as
Revised: It is defined as

Original: [Equation]
Revised: [Equation]

Original:  is the binary Morgan fingerprint vector for the molecule
Revised: , FP X is the binary Morgan fingerprint vector for the molecule

Original: [Equation]

Original: X .
Revised: X .

Original: In this paper, it is a 2048-bit binary vector.• QED represents a quantitative estimate of drug-likeness.
Revised: In this paper, it is a 2048-bit binary vector. • QED represents a quantitative estimate of drug-likeness.

Original: QED score ranges from 0 to 1.
Revised: QED score ranges from 0 to 1.

Original: Itcan be evaluated by the RDKit package ( https://www.rdkit.org/ ).
Revised: It can be evaluated by the RDKit package ( https://www.rdkit.org/ ).

Original: • SA (Synthetic Accessibility) score measures how hard it is to synthesize a given molecule,based on a combination of the molecule’s fragments contributions [61].
Revised: • SA (Synthetic Accessibility) score measures how hard it is to synthesize a given molecule, based on a combination of the molecule’s fragments contributions [61].

Original: It is evaluated via RDKit [60].
Revised: It is evaluated via RDKit [60].

Original: The raw SA score ranges from 1 to 10.
Revised: The raw SA score ranges from 1 to 10.

Original: A higher SA score means the moleculeis hard to be synthesized and is not desirable.
Revised: A higher SA score means the molecule is hard to be synthesized and is not desirable.

Original: • Run Time .
Revised: • Run Time .

Original: Unlike optimizing some simple oracles such as QED and LogP scores, the docking simulation need to search 3D molecular conformation docked in the target, which is computationally expensive.
Revised: Unlike optimizing some simple oracles such as QED and LogP scores, the docking simulation need to search 3D molecular conformation docked in the target, which is computationally expensive.

Original: Thus run time is an important metric to measure the efficiencyof the methods.
Revised: Thus run time is an important metric to measure the efficiency of the methods.

Original: E Implementation DetailsE.1 Software/Hardware Configuration
Revised: E Implementation Details E.1 Software/Hardware Configuration

Original: We implemented RGA using Pytorch 1.10.2, Python 3.7, RDKit v2020.09.1.0 on an Intel Xeon
Revised: We implemented RGA using Pytorch 1.10.2, Python 3.7, RDKit v2020.09.1.0 on an Intel Xeon

Original: E5-2690 machine with 256G RAM and NVIDIA Pascal Titan X GPUs.
Revised: E5-2690 machine with 256G RAM and NVIDIA Pascal Titan X GPUs.

Original: E.2 Hyperparameter Setup
Revised: E.2 Hyperparameter Setup

Original: The neural architectures of policy networks are E(3)-equivariant neural network (ENN) [18].
Revised: The neural architectures of policy networks are E(3)-equivariant neural network (ENN) [19].

Original: Thevocabulary set V = { C, N, O, S, H, other } .
Revised: The vocabulary set V = { C, N, O, S, H, other } .

Original: In ENN, the number of layers is set to 3, i.e., L = 3 ;the hidden dimension is set to 100, i.e., d = 100 .
Revised: In ENN, the number of layers is set to 3, i.e., L = 3 ; the hidden dimension is set to 100, i.e., d = 100 .

Original: In Equation 1, MLP e ( · ) , MLP x ( · ) , MLP h ( · ) aretwo-layer MLP in ENN with Swish activation [50] in hidden layer.
Revised: In Equation 1, MLP e ( · ) , MLP x ( · ) , MLP h ( · ) are two-layer MLP in ENN with Swish activation [51] in hidden layer.

Original: Summation function is usedas aggregation function to aggregate the last-layer’s node embedding into graph-level embedding.
Revised: Summation function is used as aggregation function to aggregate the last-layer’s node embedding into graph-level embedding.

Original: All the atoms that within the binding site are used as the input of ENN.
Revised: All the atoms that within the binding site are used as the input of ENN.

Original: REINFORCE is used toimplement policy gradient [62, 28].
Revised: REINFORCE is used to implement policy gradient [62, 29].

Original: Adam is utilized as optimizer with learning rate 0 .
Revised: Adam is utilized as optimizer with learning rate 0 .

Original: 001 for bothcrossover and mutation policy networks.
Revised: 001 for both crossover and mutation policy networks.

Original: The reaction set R is generated via merging two publicreaction libraries: (1) the AutoClickChemRxn set (36 reactions) [46] and (2) RobustRxn set (58reactions [49]).
Revised: The reaction set R is generated via merging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions) [47] and (2) RobustRxn set ( reactions [50]).

Original: We use RDKit [60] to perform in silico chemical reaction based on SMARTS reactiontemplate, then we have |R| = 36 + 58 = 94 .
Revised: We use RDKit [60] to perform in silico chemical reaction based on SMARTS reaction template, then we have |R| = 36 + 58 = 94 .

Original: In each generation, we generate up to 200 offspring(100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vina scores)ones for the next generation, i.e., K = 50 .
Revised: In each generation, we generate up to 200 offspring (100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vina scores) ones for the next generation, i.e., K = 50 .

Original: E.3 Code Repository
Revised: E.3 Code Repository

Original: The code repository is uploaded in supplementary material for reproducibility.
Revised: The code repository is uploaded in supplementary material for reproducibility.

Original: F Additional ExperimentF.1 Efficiency Study
Revised: F Additional Experiment F.1 Efficiency Study

Original: As mentioned before, unlike optimizing some simple oracles such as QED and LogP scores, thedocking simulation need to search 3D molecular conformation docked in the target, which is timeconsuming.
Revised: As mentioned before, unlike optimizing some simple oracles such as QED and LogP scores, the docking simulation need to search 3D molecular conformation docked in the target, which is timeconsuming.

Original: Thus run time is an important measurement to evaluate the efficiency of the methods.
Revised: Thus run time is an important measurement to evaluate the efficiency of the methods.

Original: We report the bar of run time over different targets for all the compared methods in Figure 4.
Revised: We report the bar of run time over different targets for all the compared methods in Figure 4.

Original: Therun times varies greatly over different methods.
Revised: The run times varies greatly over different methods.

Original: Thus, for ease of visualization, we divided all themethods into two groups.
Revised: Thus, for ease of visualization, we divided all the methods into two groups.

Original: One is slow group, containing 5 methods: screening, GEGL, REINVENT,
Revised: One is slow group, containing 5 methods: screening, GEGL, REINVENT,

Original: RationaleRL and JTVAE, where all the methods take more than 10 hours.
Revised: RationaleRL and JTVAE, where all the methods take more than 10 hours.

Original: Another is fast group ( < 10hours), containing 7 methods, MARS, MolDQN, Gen3D, GA+D, GraphGA, Autogrow, and RGA.
Revised: Another is fast group ( <hours), containing 7 methods, MARS, MolDQN, Gen3D, GA+D, GraphGA, Autogrow, and RGA.

Original: We find that both Autogrow and RGA are efficient compared with other methods.
Revised: We find that both Autogrow and RGA are efficient compared with other methods.

Original: This attributes tothe unique design of genetic algorithm (both crossover and mutation operations) and the usage offilter after GA operators, as described in Section 3.1.
Revised: This attributes to the unique design of genetic algorithm (both crossover and mutation operations) and the usage of filter after GA operators, as described in Section 3.1.

Original: RGA is only slightly slower than Autogrowbecause it requires additional computation to pretrain/train the policy neural networks.
Revised: RGA is only slightly slower than Autogrow because it requires additional computation to pretrain/train the policy neural networks.

Original: F.2 Pretraining Equivariant neural network
Revised: F.2 Pretraining Equivariant neural network

Original: Pretraining equivariant neural network is a crucial step to RGA .
Revised: Pretraining equivariant neural network is a crucial step to RGA .

Original: We adopt CrossDocked2020 [56]dataset that contains around 22 million ligand-protein complexes as the training data for pretraining thepolicy neural networks, as mentioned in Section 3.3.
Revised: We adopt CrossDocked2020 [57] dataset that contains around 22 million ligand-protein complexes as the training data for pretraining the policy neural networks, as mentioned in Section 3.3.

Original: We split the whole dataset into training/validationdataset with ratio of 9:1.
Revised: We split the whole dataset into training/validation [Figure] dataset with ratio of 9:1.

Original: Each data point is a target-ligand complex and the binding affinity (scalar).
Revised: Each data point is a target-ligand complex and the binding affinity (scalar).

Original: We report the validation loss of ENN on target-ligand binding affinity prediction task.
Revised: We report the validation loss of ENN on target-ligand binding affinity prediction task.

Original: The validationloss function is root mean square error (RMSE).
Revised: The validation loss function is root mean square error (RMSE).

Original: We find the learning process converges rapidly whenpassing 150K data points (within an epoch) in terms of validation MSE loss.[Figure]
Revised: We find the learning process converges rapidly when passing 150K data points (within an epoch) in terms of validation RMSE loss.

Original: F.3 Additional Ablation Study
Revised: F.3 Additional Ablation Study

Original: [Figure] To further understand our model and GA process, we conduct an ablation study to investigate theimpact of each component/strategy to optimization performance.
Revised: To further understand our model and GA process, we conduct an ablation study to investigate the impact of each component/strategy to optimization performance.

Original: Specifically, we consider thefollowing four variants of RGA.
Revised: Specifically, we consider the following four variants of RGA.

Original: RGA -pretrain is a variant of RGA that does not pretrain the policyneural network.
Revised: RGA -pretrain is a variant of RGA that does not pretrain the policy neural network.

Original: RGA -KT (Knowledge Transfer) is a variant of RGA that does not training policyneural network on different target proteins, i.e., optimizing ligand for one target at a time.
Revised: RGA -KT (Knowledge Transfer) is a variant of RGA that does not training policy neural network on different target proteins, i.e., optimizing ligand for one target at a time.

Original: RGA -MU(mutation) is a variant of RGA that does not involve mutation operation in GA.
Revised: RGA -MU (mutation) is a variant of RGA that does not involve mutation operation in GA.

Original: That is, all the ligandsare generated via crossover operator.
Revised: That is, all the ligands are generated via crossover operator.

Original: Correspondingly, RGA -CO (crossover) is a variant that doesnot use crossover operation in GA, which means no mutation operator.
Revised: Correspondingly, RGA -CO (crossover) is a variant that does not use crossover operation in GA, which means no mutation operator.

Original: The results are reportedin Table 3.
Revised: The results are reported in Table 3.

Original: We find that removing either component/strategy will cause a drop in optimizationperformance (i.e., increase in TOP-100/10/1 scores).
Revised: We find that removing either component/strategy will cause a drop in optimization performance (i.e., increase in TOP-100/10/1 scores).

Original: Both crossover and mutation are critical tothe optimization performance.
Revised: Both crossover and mutation are critical to the optimization performance.

Original: Also, both pretraining the policy networks and knowledge transferbetween different target have positive contribution to the performance. The ablation study furtherlyvalidates the effectiveness of the proposed RGA method. affinity prediction.
Revised: Also, both pretraining the policy networks and knowledge transfer affinity prediction.

Original: We plot the root-mean-square error (RMSE) loss as a function of number of passed data.
Revised: We plot the root-mean-square error (RMSE) loss as a function of number of passed data.

Original: We use early stop strategy to terminate the learning process earlier when the validation loss would not decrease to save computational resource and avoid overfitting.
Revised: We use early stop strategy to terminate the learning process earlier when the validation loss would not decrease to save computational resource and avoid overfitting.

Original: We found the learning process would converges when passing 150K data samples (within an epoch), RMSE loss decreases from more than 8 to less than 1.
Revised: We found the learning process would converges when passing 150K data samples (within an epoch), RMSE loss decreases from more than 8 to less than 1.

Original: [Table]
Revised: [Table]

Revised: between different target have positive contribution to the performance.

Revised: The ablation study furtherly validates the effectiveness of the proposed RGA method.

Original: F.4 Scalability
Revised: F.4 Scalability

Original: As mentioned before, the population size in RGA is K .
Revised: As mentioned before, the population size in RGA is K .

Original: Then we analyze the computational complexitywithin each generation.
Revised: Then we analyze the computational complexity within each generation.

Original: There are two operations, including crossover and mutation operations, asdescribed in Section 3.3.
Revised: There are two operations, including crossover and mutation operations, as described in Section 3.3.

Original: For crossover, the first step is to select the first parent molecule, we need to evaluate the probabilityover all the molecules in current population, whose complexity is O ( K ) .
Revised: For crossover, the first step is to select the first parent molecule, we need to evaluate the probability over all the molecules in current population, whose complexity is O ( K ) .

Original: The second step is to selectthe second parent molecule based on the first parent molecule, we need to evaluate the probabilityover all the remaining molecules in current population, as shown in Equation (2), whose complexityis O
Revised: The second step is to select the second parent molecule based on the first parent molecule, we need to evaluate the probability over all the remaining molecules in current population, as shown in Equation (2), whose complexity is O

Original: ( K − 1) .
Revised: ( K − 1) .

Original: The complexity of crossover operation is O ( K ) .
Revised: The complexity of crossover operation is O ( K ) .

Original: On the other hand, for mutation operation, the first step is to select the parent molecule (onlyone parent) via evaluating the probability over all the molecules in the current population, i.e.,p (1) mutation ( X parentmutation |S ( t ) ) , whose complexity is O ( K ) .
Revised: On the other hand, for mutation operation, the first step is to select the parent molecule (only one parent) via evaluating the probability over all the molecules in the current population, i.e., p (1) mutation ( X parentmutation |S ( t ) ) , whose complexity is O ( K ) .

Original: The second step is to select a mutated molecules(generated by chemical reaction), as shown in Equation (3).
Revised: The second step is to select a mutated molecules (generated by chemical reaction), as shown in Equation (3).

Original: The complexity is O ( |R| ) , where R isthe reaction set (see Table 2).
Revised: The complexity is O ( |R| ) , where R is the reaction set (see Table 2).

Original: The complexity of mutation operation is O ( |R| + K ) .
Revised: The complexity of mutation operation is O ( |R| + K ) .

Revised: [Table]

Original: To summarize, the complexity of RGA is O
Revised: To summarize, the complexity of RGA is O

Original: ( K + |R| ) .
Revised: ( K + |R| ) .

Original: That is, RGA scales linearly in population size
Revised: That is, RGA scales linearly in population size

Original: K and the size of the chemical reaction set |R| .
Revised: K and the size of the chemical reaction set |R| .

Original: As mentioned in Section E.2, |R| = 94 , K = 50 .
Revised: As mentioned in Section E.2, |R| = 94 , K = 50 .

Original: Thus, RGA owns desired scalability and is not computational expensive.
Revised: Thus, RGA owns desired scalability and is not computational expensive.

Original: [Table]

Original: F.5 Significance Studies
Revised: F.5 Significance Studies

Revised: In this section

Original: [Table]
Revised: [Table]

Original: In this section, we present the significance studies.
Revised: we present the significance studies.

Original: Specifically, we conduct the hypothesis testingto show the statistical significance of our method over the other GA methods.
Revised: Specifically [Table] we conduct the hypothesis testing to show the statistical significance of our method over the other GA methods.

Original: We compare thesignificance of the improvement over other GA methods (including AutoGrow4, Graph-GA, GA+D,and GEGL) via running 5 independent trials with different random seeds and thep-value .
Revised: We compare the significance of the improvement over other GA methods (including AutoGrow4, Graph-GA, GA+D, and GEGL) via running 5 independent trials with different random seeds and then evaluating the p-value .

Original: We consider the top-100 and top-10 score as the major metrics, which are the most importantmetrics for optimization performance.
Revised: We consider the top-100 and top-10 score as the major metrics, which are the most important metrics for optimization performance.

Original: We compare RGA versus AutoGrow 4, Graph-GA and GA+D.
Revised: We compare RGA versus AutoGrow 4, Graph-GA and GA+D.

Original: The results (p-value) are shown in Table 4.
Revised: The results (p-value) are shown in Table 4.

Original: We find almost all the p-values are less than 0.05 (exceptone value), which indicates that the improvements of RGA over other GA methods are statisticallysignificant.
Revised: We find almost all the p-values are less than 0.05 (except one value), which indicates that the improvements of RGA over other GA methods are statistically significant.

Original: F.6 Example of the generated ligand
Revised: F.6 Example of the generated ligand

Original: This section shows some examples of the generated ligands with desirable binding affinity for varioustarget proteins in Figure 6, 7, 8 and 9, respectively.
Revised: This section shows some examples of the generated ligands with desirable binding affinity for various target proteins in Figure 6, 7, 8 and 9, respectively.

Original: We observe that the generated ligands bind tightlywith the target proteins.
Revised: We observe that the generated ligands bind tightly with the target proteins.

Original: G Additional Discussion on Related Work
Revised: G Additional Discussion on Related Work

Original: Methodology .
Revised: Methodology .

Original: The molecule generations methods can be divided into two categories.
Revised: The molecule generations methods can be divided into two categories.

Original: The firstone is deep generative models (DGMs), which leverage the continuous representation to estimatethe data distribution using various kinds of deep neural networks, including variational autoencoder(VAE) [19, 20], generative adversarial network (GAN) [21, 22], normalizing flow model [23, 63, 24],energy based model [25] and diffusion model [64], etc.
Revised: The first one is deep generative models (DGMs), which leverage the continuous representation to estimate the data distribution using various kinds of deep neural networks, including variational autoencoder (VAE) [20, 21], generative adversarial network (GAN) [22, 23], normalizing flow model [24, 63, 25], energy based model [26] and diffusion model [64], etc.

Original: The second one is combinatorial optimizationmethods, which directly search the discrete chemical space, including genetic algorithm (GA) [15, 26,27], reinforcement learning approaches (RL) [28, 29, 30, 31], Bayesian Optimization (BO) [33, 65],
Revised: The second one is combinatorial optimization methods, which directly search the discrete chemical space, including genetic algorithm (GA) [16, 27, 28], reinforcement learning approaches (RL) [29, 30, 31, 32], Bayesian Optimization (BO) [34, 65],

Original: Monte Carlo Tree Search (MCTS) [66, 67, 31] and Markov Chain Monte Carlo (MCMC) [34, 35, 36].
Revised: Monte Carlo Tree Search (MCTS) [66, 67, 32] and Markov Chain Monte Carlo (MCMC) [35, 36, 37].

Original: Deep generative models (DGMs) usually require a large amount of data to fit, which impedes theirusage in low data regime.
Revised: Deep generative models (DGMs) usually require a large amount of data to fit, which impedes their usage in low data regime.

Original: In contrast, combinatorial optimization methods require less training data,while the trade-off is the need to call the optimization oracles during the exploration in the chemicalspace [30, 37, 48, 27, 13].
Revised: In contrast, combinatorial optimization methods require less training data, while the trade-off is the need to call the optimization oracles during the exploration in the chemical space [31, 38, 49, 28, 13].

Original: Among all the machine learning methods, Genetic algorithm (GA) exhibits superior performance insome standard benchmarks [13, 14].
Revised: Among all the machine learning methods, Genetic algorithm (GA) exhibits superior performance in some standard benchmarks [13, 14, 15].

Original: The key reason is GA’s global assembling strategy.
Revised: The key reason is GA’s global assembling strategy.

Original: Specifically,in each generation (iteration), GA maintains a population of molecule candidates (a.k.a. parents), andconducts the crossover operation between two (random-selected) parent candidates, which enablesrelatively large exchanges on molecular sub-graph between molecular graphs.
Revised: Specifically, in each generation (iteration), GA maintains a population of molecule candidates (a.k.a. parents), and conducts the crossover operation between two (random-selected) parent candidates, which enables relatively large exchanges on molecular sub-graph between molecular graphs.

Original: However, GA isleveraging random-walk based mutation and crossover operations [15, 16] and is essentially based onbrute-force trial and error. 
Revised: However, GA is leveraging random-walk based mutation and crossover operations [16, 17] and is essentially based on brute-force trial and error. [Figure]

Original: On the other hand, Reinforcement learning (RL) methods are good at navigating the discrete space viaprioritizing the promising searching branches and circumventing brute-force search.
Revised: On the other hand, Reinforcement learning (RL) methods are good at navigating the discrete space via prioritizing the promising searching branches and circumventing brute-force search.

Original: The current RLbased methods [28, 29, 30, 31] slightly left behind other state-of-the-art combinatorial optimizationmethods [37, 14].
Revised: The current RLbased methods [29, 30, 31, 32] slightly left behind other state-of-the-art combinatorial optimization methods [38, 14].

Original: The main reason is that current RL based molecule optimization approaches arebased on auto-regressive assembling strategy, i.e., growing molecules iteratively via adding a basic[Figure] building block one time, where the building block can be either a token in SMILES representation [28]or a substructure in molecular graph representation [29, 30, 31].
Revised: The main reason is that current RL based molecule optimization approaches are based on auto-regressive assembling strategy, i.e., growing molecules iteratively via adding a basic building block one time, where the building block can be either a token in SMILES representation [29] or a substructure in molecular graph representation [30, 31, 32].

Original: Such assembling strategy areessentially local search methods, which hinders the algorithm’s ability to overcome the roughoptimization landscape (or energy barrier) and is easy to be stuck in the local optimum [68, 69].
Revised: Such assembling strategy are essentially local search methods, which hinders the algorithm’s ability to overcome the rough optimization landscape (or energy barrier) and is easy to be stuck in the local optimum [68, 69].

Original: Discussion .
Revised: Discussion .

Original: Among all the machine learning methods, molecular graph level genetic algorithm (GA)exhibits state-of-the-art performance in some standard molecule optimization benchmarks [13, 14].
Revised: Among all the machine learning methods, molecular graph level genetic algorithm (GA) exhibits state-of-the-art performance in some standard molecule optimization benchmarks [13, 14, 15]. [Figure]

Original: The key reason is GA’s assembling manner.
Revised: The key reason is GA’s assembling manner.

Original: Specifically, in each generation (iteration), GA maintainsa population of possible candidates (a.k.a. parents), and conducts the crossover operation betweentwo candidates to generate new offspring, which enables thorough exploration to the chemical space.
Revised: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a. parents), and conducts the crossover operation between two candidates to generate new offspring, which enables thorough exploration to the chemical space.

Original: However, there is still improvement space for GA.
Revised: However, there is still improvement space for GA.

Original: GA are leveraging random-walk based mutationand crossover operations [15] and suffers from brute-force trial and error strategy.
Revised: GA are leveraging random-walk based mutation and crossover operations [16] and suffers from brute-force trial and error strategy.

Revised: On the other hand, reinforcement learning approaches are good at navigating the discrete space via prioritizing the promising decisions that are worth investigating, for example, AlphaGo successfully

Original: [Figure]
Revised: [Figure]

Original: On the other hand, reinforcement learning approaches are good at navigating the discrete space viaprioritizing the promising decisions that are worth investigating, for example, AlphaGo successfullyapplied RL to defeat a professional human Go player [70].
Revised: applied RL to defeat a professional human Go player [70].

Original: However, the current RL based drugdesign methods [28, 29, 30] slightly left behind other state-of-the-art combinatorial optimizationmethods.
Revised: However, the current RL based drug design methods [29, 30, 31] slightly left behind other state-of-the-art combinatorial optimization methods.

Original: The main reason lies at the inferior assembling strategy, which grows molecule in anauto-regressive fashion.
Revised: The main reason lies at the inferior assembling strategy, which grows molecule in an auto-regressive fashion.

Original: It is hard for this kind of local search strategy to overcome the barrier of theobjective, so it is easy to be trapped into the local optimum.
Revised: It is hard for this kind of local search strategy to overcome the barrier of the objective, so it is easy to be trapped into the local optimum.

Original: [Figure]

Original: Deep learning methods can also enhance genetic algorithm.
Revised: Deep learning methods can also enhance genetic algorithm.

Original: Due to the random selection used ingenetic algorithm, it is challenging to apply deep learning methods in the generation of new candidates(molecules in this paper).
Revised: Due to the random selection used in genetic algorithm, it is challenging to apply deep learning methods in the generation of new candidates (molecules in this paper).

Original: Deep learning can be used to compose fitness evaluation to select theoffspring.
Revised: Deep learning can be used to compose fitness evaluation to select the offspring.

Original: For example, GA+D
Revised: For example, GA+D

Original: [26] leverages deep neural network as a discriminator to measure thedrug’s proximity to the training data, which is incorporated as a scorer in fitness evaluation.
Revised: [27] leverages deep neural network as a discriminator to measure the drug’s proximity to the training data, which is incorporated as a scorer in fitness evaluation.

Original: [71] traina deep neural network-based property predictor and leverage it to enhance the evolutionary algorithm.
Revised: [71] train a deep neural network-based property predictor and leverage it to enhance the evolutionary algorithm.

Original: In this paper, we attempt to enhance genetic algorithm using reinforcement learning technique.
Revised: In this paper, we attempt to enhance genetic algorithm using reinforcement learning technique.

Original: Specifically, we propose Reinforced Genetic Algorithm ( RGA ), which inherits the assembling mannerfrom genetic algorithm and use reinforcement learning to guide the search over the chemical space.[32] also combine RL and GA, which uses LSTM (guided by RL agent) to imitate GA process,however, it is unable to inherit the GA’s flexible assembling manner due to the auto-regressive essenceof LSTM.
Revised: Specifically, we propose Reinforced Genetic Algorithm ( RGA ), which inherits the assembling manner from genetic algorithm and use reinforcement learning to guide the search over the chemical space. [33] also combine RL and GA, which uses LSTM (guided by RL agent) to imitate GA process, however, it is unable to inherit the GA’s flexible assembling manner due to the auto-regressive essence of LSTM.

Original: [Figure]
Revised: [Figure]