Original: Reinforced Genetic Algorithm for Structure-based Drug Design
Revised: Reinforced Genetic Algorithm for Structure-based Drug Design

Original: Abstract
Revised: Abstract

Original: Structure-based drug design (SBDD) aims to discover drug candidates by findingmolecules (ligands) that bind tightly to a disease-related protein (targets), which isthe primary approach to computer-aided drug discovery.
Revised: Structure-based drug design (SBDD) aims to discover drug candidates by findingmolecules (ligands) that bind tightly to a disease-related protein (targets), which isthe primary approach to computer-aided drug discovery.

Original: Recently, applying deepgenerative models for three-dimensional (3D) molecular design conditioned onprotein pockets to solve SBDD has attracted much attention, but their formulationas probabilistic modeling often leads to unsatisfactory optimization performance.
Revised: Recently, applying deepgenerative models for three-dimensional (3D) molecular design conditioned onprotein pockets to solve SBDD has attracted much attention, but their formulationas probabilistic modeling often leads to unsatisfactory optimization performance.

Original: On the other hand, traditional combinatorial optimization methods such as geneticalgorithms (GA) have demonstrated state-of-the-art performance in various molecular optimization tasks.
Revised: On the other hand, traditional combinatorial optimization methods such as geneticalgorithms (GA) have demonstrated state-of-the-art performance in various molecular optimization tasks.

Original: However, they do not utilize protein target structure toinform design steps but rely on a random-walk-like exploration, which leads tounstable performance and no knowledge transfer between different tasks despitethe similar binding physics.
Revised: However, they do not utilize protein target structure toinform design steps but rely on a random-walk-like exploration, which leads tounstable performance and no knowledge transfer between different tasks despitethe similar binding physics.

Original: To achieve a more stable and efficient SBDD, wepropose Reinforced Genetic Algorithm ( RGA ) that uses neural models to prioritizethe profitable design steps and suppress random-walk behavior.
Revised: To achieve a more stable and efficient SBDD, wepropose Reinforced Genetic Algorithm ( RGA ) that uses neural models to prioritizethe profitable design steps and suppress random-walk behavior.

Original: The neural modelstake the 3D structure of the targets and ligands as inputs and are pre-trained usingnative complex structures to utilize the knowledge of the shared binding physicsfrom different targets and then fine-tuned during optimization.
Revised: The neural modelstake the 3D structure of the targets and ligands as inputs and are pre-trained usingnative complex structures to utilize the knowledge of the shared binding physicsfrom different targets and then fine-tuned during optimization.

Original: We conduct thorough empirical studies on optimizing binding affinity to various disease targetsand show that RGA outperforms the baselines in terms of docking scores and ismore robust to random initializations.
Revised: We conduct thorough empirical studies on optimizing binding affinity to various disease targetsand show that RGA outperforms the baselines in terms of docking scores and ismore robust to random initializations.

Original: The ablation study also indicates that thetraining on different targets helps improve the performance by leveraging the sharedunderlying physics of the binding processes.
Revised: The ablation study also indicates that thetraining on different targets helps improve the performance by leveraging the sharedunderlying physics of the binding processes.

Original:  Introduction
Revised:  Introduction

Original: Rapid drug discovery that requires less time and cost is of significant interest in pharmaceuticalscience.
Revised: Rapid drug discovery that requires less time and cost is of significant interest in pharmaceuticalscience.

Original: Structure-based drug design (SBDD) [1] that leverages the three-dimensional (3D) structuresof the disease-related proteins to design drug candidates is one primary approach to accelerate thedrug discovery processes with physical simulation and data-driven modeling.
Revised: Structure-based drug design (SBDD) [1] that leverages the three-dimensional (3D) structuresof the disease-related proteins to design drug candidates is one primary approach to accelerate thedrug discovery processes with physical simulation and data-driven modeling.

Original: According to the lockand key model [2], the molecules that bind tighter to a disease target are more likely to exposebioactivity against the disease, which has been verified experimentally [3].
Revised: According to the lockand key model [2], the molecules that bind tighter to a disease target are more likely to exposebioactivity against the disease, which has been verified experimentally [3].

Original: As AlphaFold2 hasprovided accurate predictions to most human proteins [4, 5], SBDD has a tremendous opportunity todiscover new drugs for new targets that we cannot model before [6].
Revised: As AlphaFold2 hasprovided accurate predictions to most human proteins [4, 5], SBDD has a tremendous opportunity todiscover new drugs for new targets that we cannot model before [6].

Original: SBDD could be formulated as an optimization problem where the objective function is the bindingaffinity estimated by simulations such as docking [2].
Revised: SBDD could be formulated as an optimization problem where the objective function is the bindingaffinity estimated by simulations such as docking [2].

Original: The most widely used design method is virtualscreening, which exhaustively investigates every molecule in a library and ranks them.
Revised: The most widely used design method is virtualscreening, which exhaustively investigates every molecule in a library and ranks them.

Original: Lyu et al.successfully discovered new chemotypes for AmpC β -lactamase and the D 4 dopamine receptor bystudying hundreds of millions of molecules with docking simulation [7].
Revised: Lyu et al.successfully discovered new chemotypes for AmpC β -lactamase and the D 4 dopamine receptor bystudying hundreds of millions of molecules with docking simulation [7].

Original: However, the number of the drug-like molecules is large as estimated to be 10 60 [1], and it is computationally prohibitive toscreen all of the possible molecules.
Revised: However, the number of the drug-like molecules is large as estimated to be 10 60 [1], and it is computationally prohibitive toscreen all of the possible molecules.

Original: Though machine learning approaches have been developed toaccelerate screening [8, 9], it is still challenging to screen large enough chemical space within theforeseeable future.
Revised: Though machine learning approaches have been developed toaccelerate screening [8, 9], it is still challenging to screen large enough chemical space within theforeseeable future.

Original: Instead of naively screening a library, designing drug candidates with generative models has beenhighlighted as a promising strategy, exemplified by [10, 11].
Revised: Instead of naively screening a library, designing drug candidates with generative models has beenhighlighted as a promising strategy, exemplified by [10, 11].

Original: This class of methods models theproblem as the generation of ligands conditioned on the protein pockets.
Revised: This class of methods models theproblem as the generation of ligands conditioned on the protein pockets.

Original: However, as generativemodels are trained to learn the distribution of known active compounds, they tend to producemolecules similar to training data [12], which discourages finding novel molecules and leads tounsatisfactory optimization performance.
Revised: However, as generativemodels are trained to learn the distribution of known active compounds, they tend to producemolecules similar to training data [12], which discourages finding novel molecules and leads tounsatisfactory optimization performance.

Original: A more straightforward solution is a combinatorial optimization algorithm that searches the implicitlydefined discrete chemical space.
Revised: A more straightforward solution is a combinatorial optimization algorithm that searches the implicitlydefined discrete chemical space.

Original: As shown in multiple standard molecule optimization benchmarks [13, 14], combinatorial optimization methods, especially genetic algorithms (GA)
Revised: As shown in multiple standard molecule optimization benchmarks [13, 14], combinatorial optimization methods, especially genetic algorithms (GA)

Original: [15, 16],often perform better than deep generative models.
Revised: [15, 16],often perform better than deep generative models.

Original: The key to superior performance is GA’s actiondefinition.
Revised: The key to superior performance is GA’s actiondefinition.

Original: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a.
Revised: Specifically, in each generation (iteration), GA maintains a population of possible candidates (a.k.a.

Original: parents) and conducts the crossover between two candidates and mutation from a singlecandidate to generate new offspring.
Revised: parents) and conducts the crossover between two candidates and mutation from a singlecandidate to generate new offspring.

Original: These two types of actions, crossover and mutation, enableglobal and local traversal over the chemical space, allowing a thorough exploration and superioroptimization performance.
Revised: These two types of actions, crossover and mutation, enableglobal and local traversal over the chemical space, allowing a thorough exploration and superioroptimization performance.

Original: However, most GAs select mutation and crossover operations randomly [15], leading to significantvariance between independent runs.
Revised: However, most GAs select mutation and crossover operations randomly [15], leading to significantvariance between independent runs.

Original: Especially in SBDD, when the oracle functions are expensivemolecular simulations, it is resource-consuming to ensure stability by running multiple times.
Revised: Especially in SBDD, when the oracle functions are expensivemolecular simulations, it is resource-consuming to ensure stability by running multiple times.

Original: Further,most current combinatorial methods are designed for general-purpose molecular optimization andsimply use a docking simulation as an oracle.
Revised: Further,most current combinatorial methods are designed for general-purpose molecular optimization andsimply use a docking simulation as an oracle.

Original: It is challenging to leverage the structure of proteins inthese methods, and we need to start from scratch whenever we change a protein target, even thoughthe physics of ligand-protein interaction is shared.
Revised: It is challenging to leverage the structure of proteins inthese methods, and we need to start from scratch whenever we change a protein target, even thoughthe physics of ligand-protein interaction is shared.

Original: Ignoring the shared information across tasksleads to unnecessary exploration steps and, thus, demands for many more oracle calls, which requireexpensive and unnecessary simulations [17].
Revised: Ignoring the shared information across tasksleads to unnecessary exploration steps and, thus, demands for many more oracle calls, which requireexpensive and unnecessary simulations [17].

Original: To overcome these issues in the GA method, we propose Reinforced Genetic Algorithm ( RGA ),which attempts to reformulate an evolutionary process as a Markov decision process and uses neuralnetworks to make informed decisions and suppress the random-walk behavior.
Revised: To overcome these issues in the GA method, we propose Reinforced Genetic Algorithm ( RGA ),which attempts to reformulate an evolutionary process as a Markov decision process and uses neuralnetworks to make informed decisions and suppress the random-walk behavior.

Original: Specifically, weutilize an E(3)-equivariant neural network [18] to choose parents and mutation types based on the3D structure of the ligands and proteins.
Revised: Specifically, weutilize an E(3)-equivariant neural network [18] to choose parents and mutation types based on the3D structure of the ligands and proteins.

Original: The networks are pre-trained with various native complexstructures to utilize the knowledge of the shared binding physics between different targets and thenfine-tuned with a reinforcement learning algorithm during optimizations.
Revised: The networks are pre-trained with various native complexstructures to utilize the knowledge of the shared binding physics between different targets and thenfine-tuned with a reinforcement learning algorithm during optimizations.

Original: We test RGA ’s performancewith various disease-related targets, including the main protease of SARS-CoV-2.
Revised: We test RGA ’s performancewith various disease-related targets, including the main protease of SARS-CoV-2.

Original: The main contributions of this paper can be summarized as follows:
Revised: The main contributions of this paper can be summarized as follows:

Original: • We propose an evolutionary Markov decision process (EMDP) that reformulates an evolutionaryprocess as a Markov decision process, where the state is a population of molecules instead of asingle molecule (Section 3.2).• We show the first successful attempt to use a neural model to guide the crossover and mutationoperations in a genetic algorithm to suppress random-walk behavior and explore the chemicalspace intelligently (Section 3.3).• We present a structure-based de novo drug design algorithm that outperforms baseline methodsconsistently through thorough empirical studies on optimizing binding affinity by leveraging theunderlying binding physics (Section 4).
Revised: • We propose an evolutionary Markov decision process (EMDP) that reformulates an evolutionaryprocess as a Markov decision process, where the state is a population of molecules instead of asingle molecule (Section 3.2).• We show the first successful attempt to use a neural model to guide the crossover and mutationoperations in a genetic algorithm to suppress random-walk behavior and explore the chemicalspace intelligently (Section 3.3).• We present a structure-based de novo drug design algorithm that outperforms baseline methodsconsistently through thorough empirical studies on optimizing binding affinity by leveraging theunderlying binding physics (Section 4).

Original:  Related Works
Revised:  Related Works

Original: We will discuss the related works on methods of drug design and discuss the advantage of theproposed method over the existing works.
Revised: We will discuss the related works on methods of drug design and discuss the advantage of theproposed method over the existing works.

Original: General Molecular Design .
Revised: General Molecular Design .

Original: Molecular generation methods offer a promising direction for theautomated design of molecules with desired pharmaceutical properties such as synthesis accessibilityand drug-likeliness.
Revised: Molecular generation methods offer a promising direction for theautomated design of molecules with desired pharmaceutical properties such as synthesis accessibilityand drug-likeliness.

Original: Based on how to generate or search molecules, these approaches can be categorized into two types, (1) deep generative models (DGMs) imitate the molecular data distribution,including variational autoencoder (VAE)
Revised: Based on how to generate or search molecules, these approaches can be categorized into two types, (1) deep generative models (DGMs) imitate the molecular data distribution,including variational autoencoder (VAE)

Original: [19, 20], generative adversarial network (GAN) [21, 22],normalizing flow model [23, 24], energy based model [25]; and (2) combinatorial optimizationmethods directly search over the discrete chemical space, including genetic algorithm (GA) [15, 26,27], reinforcement learning approaches (RL) [28, 29, 30, 31, 32], Bayesian optimization (BO) [33],
Revised: [19, 20], generative adversarial network (GAN) [21, 22],normalizing flow model [23, 24], energy based model [25]; and (2) combinatorial optimizationmethods directly search over the discrete chemical space, including genetic algorithm (GA) [15, 26,27], reinforcement learning approaches (RL) [28, 29, 30, 31, 32], Bayesian optimization (BO) [33],

Original: Markov chain Monte Carlo (MCMC) [34, 35, 36] and gradient ascent [37, 38].
Revised: Markov chain Monte Carlo (MCMC) [34, 35, 36] and gradient ascent [37, 38].

Original: General molecular design algorithms often use general black-box oracle functions, and some areonly tested with trivial or self-designed oracles.
Revised: General molecular design algorithms often use general black-box oracle functions, and some areonly tested with trivial or self-designed oracles.

Original: For example, using penalized octanol-water partitioncoefficient (LogP) as the oracle function, it grows monotonically with the number of carbons, andthus there exists a trivial policy to optimize LogP. These oracles do not reflect the challenges of realdrug discovery, and those algorithms have limited value for pharmaceutical discovery.
Revised: For example, using penalized octanol-water partitioncoefficient (LogP) as the oracle function, it grows monotonically with the number of carbons, andthus there exists a trivial policy to optimize LogP. These oracles do not reflect the challenges of realdrug discovery, and those algorithms have limited value for pharmaceutical discovery.

Original: Recent worksare optimizing docking scores to simulate a more realistic discovery scenario [39, 40, 17, 41], sameas our work.
Revised: Recent worksare optimizing docking scores to simulate a more realistic discovery scenario [39, 40, 17, 41], sameas our work.

Original: However, they are still ignoring the information in the given protein structures thatcould potentially accelerate the design process.
Revised: However, they are still ignoring the information in the given protein structures thatcould potentially accelerate the design process.

Original: However, the extension to leveraging the structuralknowledge is nontrivial.
Revised: However, the extension to leveraging the structuralknowledge is nontrivial.

Original: Structure-based Drug Design .
Revised: Structure-based Drug Design .

Original: Structure-based drug design (SBDD) could utilize the structuralinformation to guide the design of molecules, which are potentially more efficient in drug discoverytasks but poses additional challenges of how to leverage the structures.
Revised: Structure-based drug design (SBDD) could utilize the structuralinformation to guide the design of molecules, which are potentially more efficient in drug discoverytasks but poses additional challenges of how to leverage the structures.

Original: Since early 1990s, various
Revised: Since early 1990s, various

Original: SBDD algorithms have been proposed, mostly based on combinatorial optimization algorithms suchas tree search [42, 43, 44] and evolutionary algorithms [45, 46].
Revised: SBDD algorithms have been proposed, mostly based on combinatorial optimization algorithms suchas tree search [42, 43, 44] and evolutionary algorithms [45, 46].

Original: Those methods typically optimizethe ligands in the pockets according to a physical model characterizing the binding affinity.
Revised: Those methods typically optimizethe ligands in the pockets according to a physical model characterizing the binding affinity.

Original: Forexample, RASSE [42] used a force-field-like scoring function [47] to evaluate the partial solutionswithin a tree search.
Revised: Forexample, RASSE [42] used a force-field-like scoring function [47] to evaluate the partial solutionswithin a tree search.

Original: However, obtaining a fast and accurate model to quantify binding free energyitself is still an unsolved challenge.
Revised: However, obtaining a fast and accurate model to quantify binding free energyitself is still an unsolved challenge.

Original: Recently, generative modeling of 3D molecules conditioned on protein targets is attracting moreattention [10, 11].
Revised: Recently, generative modeling of 3D molecules conditioned on protein targets is attracting moreattention [10, 11].

Original: Similar to DGMs in general molecular design, those methods learn the atom’scompositional and spatial distribution of native structure of protein-ligand complexes with neuralmodels and design new ligands by complete the complex structure given targets.
Revised: Similar to DGMs in general molecular design, those methods learn the atom’scompositional and spatial distribution of native structure of protein-ligand complexes with neuralmodels and design new ligands by complete the complex structure given targets.

Original: Deep generativemodels are end-to-end and data-driven thus surpass the necessity of understanding the physics ofinteraction.
Revised: Deep generativemodels are end-to-end and data-driven thus surpass the necessity of understanding the physics ofinteraction.

Original: However, as the training objective is to learn the distribution of known active compounds,the models tend to produce molecules close to the training set [12], which is undesired in terms ofpatentability and leads to unsatisfactory optimization performance.
Revised: However, as the training objective is to learn the distribution of known active compounds,the models tend to produce molecules close to the training set [12], which is undesired in terms ofpatentability and leads to unsatisfactory optimization performance.

Original:  Method
Revised:  Method

Original: In this paper, we focus on structure-based drug design.
Revised: In this paper, we focus on structure-based drug design.

Original: The goal is to design drug molecules(a.k.a.
Revised: The goal is to design drug molecules(a.k.a.

Original: ligands) that could bind tightly with the disease-related proteins (a.k.a. targets).
Revised: ligands) that could bind tightly with the disease-related proteins (a.k.a. targets).

Original: Giventhe 3D structures of the target proteins, including binding site information, docking is a popularcomputational method for assessing the binding affinity, which can be roughly retrieved as the freeenergy changes during the binding processes.
Revised: Giventhe 3D structures of the target proteins, including binding site information, docking is a popularcomputational method for assessing the binding affinity, which can be roughly retrieved as the freeenergy changes during the binding processes.

Original: We present a variant of genetic algorithm that is guidedby reinforcement learning and a docking oracle.
Revised: We present a variant of genetic algorithm that is guidedby reinforcement learning and a docking oracle.

Original: Next, we will first describe the general evolutionaryprocess used in genetic algorithms (Section 3.1); Then we will present how to model this evolutionaryprocess as a Markov decision process (MDP) where RL framework can be constructed (Section 3.2);
Revised: Next, we will first describe the general evolutionaryprocess used in genetic algorithms (Section 3.1); Then we will present how to model this evolutionaryprocess as a Markov decision process (MDP) where RL framework can be constructed (Section 3.2);

Original: After that, we describe the detailed implementation of this MDP framework using multiple policynetworks (Section 3.3).
Revised: After that, we describe the detailed implementation of this MDP framework using multiple policynetworks (Section 3.3).

Original: 3.1 Evolutionary Process
Revised: 3.1 Evolutionary Process

Original: In this section, we introduce the primary setting of the evolutionary processes.
Revised: In this section, we introduce the primary setting of the evolutionary processes.

Original: With both optimizationperformance and synthetic accessibility taken into account [48, 14], we follow the action settings in
Revised: With both optimizationperformance and synthetic accessibility taken into account [48, 14], we follow the action settings in

Original: Autogrow 4.0 [16].
Revised: Autogrow 4.0 [16].

Original: It demonstrated superior performance over other GA variants in the empiricalvalidation of structure-based drug design [16], and its mutation actions originated from chemicalreactions so that the designed molecules are more likely to be synthesizable.
Revised: It demonstrated superior performance over other GA variants in the empiricalvalidation of structure-based drug design [16], and its mutation actions originated from chemicalreactions so that the designed molecules are more likely to be synthesizable.

Original: Specifically, anevolutionary processes starts by randomly sampling a population of drug candidates from a library.
Revised: Specifically, anevolutionary processes starts by randomly sampling a population of drug candidates from a library.

Original: In each generation (iteration), it carries out (i) crossover between parents selected from the lastgeneration, and (ii) mutation on a single child to obtain the offspring pool.
Revised: In each generation (iteration), it carries out (i) crossover between parents selected from the lastgeneration, and (ii) mutation on a single child to obtain the offspring pool.

Original: An illustration of both crossover and mutation operations is available in Appendix.
Revised: An illustration of both crossover and mutation operations is available in Appendix.

Original: Note that we only adopted the actionsettings from Autogrow 4.0, without using other tricks such as elitism.
Revised: Note that we only adopted the actionsettings from Autogrow 4.0, without using other tricks such as elitism.

Original: Crossover , also called recombination, combines the structure of two parents to generate new children.
Revised: Crossover , also called recombination, combines the structure of two parents to generate new children.

Original: Following Autogrow 4.0 [16], we select two parents from the last generation and search for the largestcommon substructure shared between them.
Revised: Following Autogrow 4.0 [16], we select two parents from the last generation and search for the largestcommon substructure shared between them.

Original: Then we generate two children by randomly switchingtheir decorating moieties, i.e., the side chains attached to the common substructure.
Revised: Then we generate two children by randomly switchingtheir decorating moieties, i.e., the side chains attached to the common substructure.

Original: Mutation operates on a single parent molecule and modifies its structure slightly.
Revised: Mutation operates on a single parent molecule and modifies its structure slightly.

Original: Following Autogrow4.0 [16], we adopt transformations based on chemical reactions.
Revised: Following Autogrow4.0 [16], we adopt transformations based on chemical reactions.

Original: Unlike naively defined atom-editingactions, mutation steps based on chemical reactions could ensure all modification is reasonable inreality, leading to a larger probability of designing synthesizable molecules.
Revised: Unlike naively defined atom-editingactions, mutation steps based on chemical reactions could ensure all modification is reasonable inreality, leading to a larger probability of designing synthesizable molecules.

Original: We included two typesof chemical reactions: uni-molecular reactions, which only require one reactant, and bi-molecularreactions, which require two reactants.
Revised: We included two typesof chemical reactions: uni-molecular reactions, which only require one reactant, and bi-molecularreactions, which require two reactants.

Original: While uni-molecular reactions could be directly applied to theparent, we sample a purchasable compound to react with the parent when conducting a bi-molecularreaction.
Revised: While uni-molecular reactions could be directly applied to theparent, we sample a purchasable compound to react with the parent when conducting a bi-molecularreaction.

Original: In both cases, the parent serves as one reactant, and we use the main product as the childmolecule.
Revised: In both cases, the parent serves as one reactant, and we use the main product as the childmolecule.

Original: We use the chemical reactions from [16], which was originally from [46, 49].
Revised: We use the chemical reactions from [16], which was originally from [46, 49].

Original: Evolution .
Revised: Evolution .

Original: At the t -th generation (iteration), given a population of molecules denoted as S ( t ) , wegenerate an offspring pool denoted as Q ( t ) by applying crossover and mutation operations.
Revised: At the t -th generation (iteration), given a population of molecules denoted as S ( t ) , wegenerate an offspring pool denoted as Q ( t ) by applying crossover and mutation operations.

Original: Thenwe filter out the ones with undesirable physical and chemical properties (e.g., poor solubility, hightoxicity) in the offspring pool and select the most promising K to form the next generation pool( S ( t +1) ).
Revised: Thenwe filter out the ones with undesirable physical and chemical properties (e.g., poor solubility, hightoxicity) in the offspring pool and select the most promising K to form the next generation pool( S ( t +1) ).

Original: 3.2 Evolutionary Markov Decision Process
Revised: 3.2 Evolutionary Markov Decision Process

Original: Next we propose the evolutionary Markov decision process (EMDP) that formulates an evolutionaryprocess of genetic algorithm as a Markov decision process (MDP).
Revised: Next we propose the evolutionary Markov decision process (EMDP) that formulates an evolutionaryprocess of genetic algorithm as a Markov decision process (MDP).

Original: The primary purpose is to utilizereinforcement learning algorithms to train networks to inform the decision steps to replace randomselections.
Revised: The primary purpose is to utilizereinforcement learning algorithms to train networks to inform the decision steps to replace randomselections.

Original: Taking a generation as a state, Markov property that requires P ( S ( t +1)
Revised: Taking a generation as a state, Markov property that requires P ( S ( t +1)

Original: |S (1) , · · · , S ( t ) ) =
Revised: |S (1) , · · · , S ( t ) ) =

Original: P ( S ( t +1) |S ( t ) ) is naturally satisfied by the evolutionary process described above, where S ( t ) denotesthe state at the t -th generation, which is the population of ligands.
Revised: P ( S ( t +1) |S ( t ) ) is naturally satisfied by the evolutionary process described above, where S ( t ) denotesthe state at the t -th generation, which is the population of ligands.

Original: We use X to denote a ligand.
Revised: We use X to denote a ligand.

Original: Weelaborate essential components for Markov decision process as follows, and the EMDP pipeline isillustrated in Figure 1.
Revised: Weelaborate essential components for Markov decision process as follows, and the EMDP pipeline isillustrated in Figure 1.

Original: State Space .
Revised: State Space .

Original: We define the population at the t -step generation, S ( t ) , in the evolutionary process asthe state at the t -step in an EMDP.
Revised: We define the population at the t -step generation, S ( t ) , in the evolutionary process asthe state at the t -step in an EMDP.

Original: A state includes population of candidate molecules (i.e., ligand,denoted X ) and their 3D poses docked to the target, fully observable to the RL agent.
Revised: A state includes population of candidate molecules (i.e., ligand,denoted X ) and their 3D poses docked to the target, fully observable to the RL agent.

Original: At the beginningof the EMDP, we randomly select a population of candidate molecules and use docking simulation toyield their 3D poses as the initial state.
Revised: At the beginningof the EMDP, we randomly select a population of candidate molecules and use docking simulation toyield their 3D poses as the initial state.

Original: Action Space .
Revised: Action Space .

Original: The actions in an EMDP are to conduct the two evolutionary steps: crossover andmutation, in a population.
Revised: The actions in an EMDP are to conduct the two evolutionary steps: crossover andmutation, in a population.

Original: For each evolutionary step, we need two actions to conduct it.
Revised: For each evolutionary step, we need two actions to conduct it.

Original: Concretely,crossover ( X parent 1 crossover , X parent 2 crossover crossover −−−−−→ X child 1 crossover , X child 2 crossover ) can be divided to:
Revised: Concretely,crossover ( X parent 1 crossover , X parent 2 crossover crossover −−−−−→ X child 1 crossover , X child 2 crossover ) can be divided to:

Original: 1. select the first candidate ligand X parent 1 crossover from the current state (population S ( t ) );
Revised: 1. select the first candidate ligand X parent 1 crossover from the current state (population S ( t ) );

Original: 2. conditioned on the first selected candidate X parent 1 crossover
Revised: 2. conditioned on the first selected candidate X parent 1 crossover

Original: , select the second candidate ligand
Revised: , select the second candidate ligand

Original: X parent 2 crossover from
Revised: X parent 2 crossover from

Original: the remaining candidate ligand set S (
Revised: the remaining candidate ligand set S (

Original: t )
Revised: t )

Original: − { X parent
Revised: − { X parent

Original: 1 crossover } and
Revised: 1 crossover } and

Original: apply crossover
Revised: apply crossover

Original: Section 3.
Revised: Section 3.

Original: ) to them.
Revised: ) to them.

Original: Mutation ( X parentmutation mutated by ξ −−−−−−−→ X childmutation ) can be divided to:
Revised: Mutation ( X parentmutation mutated by ξ −−−−−−−→ X childmutation ) can be divided to:

Original: 1. select the candidate ligand X parentmutated to be mutated from the current state (population S ( t ) );
Revised: 1. select the candidate ligand X parentmutated to be mutated from the current state (population S ( t ) );

Original: 2. conditioned on the selected candidate ligand X parentmutated , select the reaction ξ from the reaction
Revised: 2. conditioned on the selected candidate ligand X parentmutated , select the reaction ξ from the reaction

Original: set R and apply it to X parentmutated .
Revised: set R and apply it to X parentmutated .

Original: As applying the crossover and mutation steps are deterministic, the actions in an EMDP focus onselecting parents and mutation types.
Revised: As applying the crossover and mutation steps are deterministic, the actions in an EMDP focus onselecting parents and mutation types.

Original: Upon finish the action, we could obtain the offspring pool,
Revised: Upon finish the action, we could obtain the offspring pool,

Original: Q ( t ) .
Revised: Q ( t ) .

Original: State Transition Dynamics .
Revised: State Transition Dynamics .

Original: The state transition in an EMDP is identical to the evolution inan evolutionary process.
Revised: The state transition in an EMDP is identical to the evolution inan evolutionary process.

Original: Once we finish the actions and obtain the offspring pool, Q ( t ) =
Revised: Once we finish the actions and obtain the offspring pool, Q ( t ) =

Original: { X child 1 , X child 2 , · · · } , we apply molecular quality filters to filter out the ones unlikely to be drugand then select the most promising K to form the parent set for the next generation ( S ( t +1) ).
Revised: { X child 1 , X child 2 , · · · } , we apply molecular quality filters to filter out the ones unlikely to be drugand then select the most promising K to form the parent set for the next generation ( S ( t +1) ).

Original: Reward .
Revised: Reward .

Original: We define the reward as the binding affinity change (docking score).
Revised: We define the reward as the binding affinity change (docking score).

Original: The actions leading tostronger binding score would be prioritized.
Revised: The actions leading tostronger binding score would be prioritized.

Original: As there is no “episode” concept in an EMDP, we treatevery step equally.
Revised: As there is no “episode” concept in an EMDP, we treatevery step equally.

Original: 3.3 Target-Ligand Policy Network
Revised: 3.3 Target-Ligand Policy Network

Original: To utilize molecular structures’ translational and rotational invariance, we adopt equivariance neuralnetworks (ENNs)
Revised: To utilize molecular structures’ translational and rotational invariance, we adopt equivariance neuralnetworks (ENNs)

Original: [18] as the target-ligand policy neural networks to select the actions in both mutationand crossover steps.
Revised: [18] as the target-ligand policy neural networks to select the actions in both mutationand crossover steps.

Original: Each ligand has a 3D pose that binds to the target protein, and the complexserves as the input of ENN.
Revised: Each ligand has a 3D pose that binds to the target protein, and the complexserves as the input of ENN.

Original: Specifically, we want to model a 3D graph Y , which can be ligand, target, or target-ligand complex.
Revised: Specifically, we want to model a 3D graph Y , which can be ligand, target, or target-ligand complex.

Original: The input feature can be described as Y =
Revised: The input feature can be described as Y =

Original: ( A , Z ) , where A represents atoms’ categories (thevocabulary set V =
Revised: ( A , Z ) , where A represents atoms’ categories (thevocabulary set V =

Original: { H, C, O, N, · · · } ) and Z represents 3D coordinates of the atoms.
Revised: { H, C, O, N, · · · } ) and Z represents 3D coordinates of the atoms.

Original: Suppose
Revised: Suppose

Original: D ∈ R |V|× d is the embedding matrix of all the categories of atoms in a vocabulary set V , is randomlyinitialized and learnable, d is the hidden dimension in ENN.
Revised: D ∈ R |V|× d is the embedding matrix of all the categories of atoms in a vocabulary set V , is randomlyinitialized and learnable, d is the hidden dimension in ENN.

Original: Each kind of atom corresponds to a rowin D .
Revised: Each kind of atom corresponds to a rowin D .

Original: We suppose there are N atoms, and each atom corresponds to a node in the 3D graph.
Revised: We suppose there are N atoms, and each atom corresponds to a node in the 3D graph.

Original: Nodeembeddings at the l -th layer are denoted as H ( l ) =
Revised: Nodeembeddings at the l -th layer are denoted as H ( l ) =

Original: { h ( l ) i } N i =1 , where l = 0 , 1 , · · · , L , L is numberof layers in ENN.
Revised: { h ( l ) i } N i =1 , where l = 0 , 1 , · · · , L , L is numberof layers in ENN.

Original: The initial node embedding h (0) i =
Revised: The initial node embedding h (0) i =

Original: D ⊤ a i ∈ R d embeds the i -th node, wherea i is one-hot vector that encode the category of the i -th atom.
Revised: D ⊤ a i ∈ R d embeds the i -th node, wherea i is one-hot vector that encode the category of the i -th atom.

Original: Coordinate embeddings at the l -thlayer are denoted Z ( l ) =
Revised: Coordinate embeddings at the l -thlayer are denoted Z ( l ) =

Original: { z ( l ) i } Ni =1 .
Revised: { z ( l ) i } Ni =1 .

Original: The initial coordinate embeddings Z (0) =
Revised: The initial coordinate embeddings Z (0) =

Original: { z i } Ni =1 are the real3D coordinates of all the nodes.
Revised: { z i } Ni =1 are the real3D coordinates of all the nodes.

Original: The following equation defines the feedforward rules of ENN, for i, j = 1 , · · · , N, i ̸
Revised: The following equation defines the feedforward rules of ENN, for i, j = 1 , · · · , N, i ̸

Original: = j, l = 0 , 1 , · · · , L − 1 , we have
Revised: = j, l = 0 , 1 , · · · , L − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: z ( l +1) 
Revised: l [Equation] [Equation] v ( l +1) i =

Revised: N (cid:88) j

Revised: =1 ,j ̸ = i w ( l +1) ij ∈ R d , z ( l +1)

Revised: i

Original: [Equation]
Revised: [Equation]

Original: where ⊕ denotes the concatenation of vectors; MLP e ( · ) : R 2 d +1 −→ R d ; MLP x ( · ) : R d −→
Revised: i = MLP [Equation] l [Equation]

Original:  R ; MLP h ( · ) : R 2 d −→ R d
Revised: l [Equation] where ⊕ denotes the concatenation of vectors; MLP e ( · ) : R 2 d +1 −→ R d ; MLP x ( · ) : R d −→ R ; MLP h ( · ) : R 2 d −→ R d

Original: are all two-layer multiple layer perceptrons (MLPs) with Swish activation in the hidden layer [50].
Revised: are all two-layer multiple layer perceptrons (MLPs) with Swish activation in the hidden layer [50].

Original: At the l -th layer, w ( l ) ij represents the message vector for the edge fromnode i to node j ; v ( l ) i represents the message vector for node i , z ( l ) i is the position embedding for nodei ; h ( l ) i is the node embedding for node i .
Revised: At the l -th layer, w ( l ) ij represents the message vector for the edge fromnode i to node j ; v ( l ) i represents the message vector for node i , z ( l ) i is the position embedding for nodei ; h ( l ) i is the node embedding for node i .

Original: H ( L ) =
Revised: H ( L ) =

Original: [ h ( L ) 1 , · · · , h ( L ) N ] are the node embeddings of the
Revised: [ h ( L ) 1 , · · · , h ( L ) N ] are the node embeddings of the

Original: L -th (last) layer.
Revised: L -th (last) layer.

Original: We aggregate them using sum function as readout function to obtain a representationof the 3D graph, denoted h Y .
Revised: We aggregate them using sum function as readout function to obtain a representationof the 3D graph, denoted h Y .

Original: The whole process is written as h Y = ENN ( Y ) .
Revised: The whole process is written as h Y = ENN ( Y ) .

Original: Crossover Policy Network .
Revised: Crossover Policy Network .

Original: We design two policy networks for two corresponding actions in acrossover, as mentioned in Section 3.2.
Revised: We design two policy networks for two corresponding actions in acrossover, as mentioned in Section 3.2.

Original: (1) the first action in crossover operation is to select thefirst parent ligand X parent 1 crossover from the population S ( t ) .
Revised: (1) the first action in crossover operation is to select thefirst parent ligand X parent 1 crossover from the population S ( t ) .

Original: Similar to the first action in mutation operation, we obtain a valid probability distribution over all the available ligands based on target-ligandcomplex as input feature and ENN as the neural network architecture, the selection probabilityof the ligand X parent 1 crossover ∈ S ( t ) is p (1) crossover (
Revised: Similar to the first action in mutation operation, we obtain a valid probability distribution over all the available ligands based on target-ligandcomplex as input feature and ENN as the neural network architecture, the selection probabilityof the ligand X parent 1 crossover ∈ S ( t ) is p (1) crossover (

Original: X parent 1 crossover |S ( t ) )
Revised: X parent 1 crossover |S ( t ) )

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , where Tand X denotes target and ligand (including 3D pose), respectively, T & X denotes target-ligandcomplex.
Revised: , where Tand X denotes target and ligand (including 3D pose), respectively, T & X denotes target-ligandcomplex.

Original: The second action is to select the second parent ligand conditioned on the first parent ligand selected in the first action.
Revised: The second action is to select the second parent ligand conditioned on the first parent ligand selected in the first action.

Original: Specifically, for ligand in the remaining population set, weconcatenate the ENN’s embedding of the target, first parent ligand X parent 1 crossover and the second parent ligand X parent 2 mutation , and feed it into an MLP to estimate a scalar as an unnormalized probability.
Revised: Specifically, for ligand in the remaining population set, weconcatenate the ENN’s embedding of the target, first parent ligand X parent 1 crossover and the second parent ligand X childmutation , and feed it into an MLP to estimate a scalar as an unnormalized probability.

Original: The unnormalized probabilities for all the ligands in the remaining population set are normalizedvia softmax function, i.e., p (2) crossover ( X parent 2 crossover | X parent 1 crossover , S ( t ) )
Revised: The unnormalized probabilities for all the ligands in the remaining population set are normalizedvia softmax function, i.e., p (2) crossover ( X parent 2 crossover | X parent 1 crossover , S ( t ) )

Original: =
Revised: =

Original: Softmax (cid:8) MLP ( h T ⊕ h X parent1crossover ⊕h X parent2crossover ) , · · · , (cid:9)
Revised: Softmax (cid:8) MLP ( h T ⊕ h X parent1crossover ⊕h X parent2crossover ) , · · · , (cid:9)

Original: X parent2crossover ∈S ( t ) −{ X parent1crossover } .
Revised: X parent2crossover ∈S ( t ) −{ X parent1crossover } .

Original: Given two parents ligands, crossover finds the largest substructure that the two parent compounds share and generates a child by combining their decoratingmoieties.
Revised: Given two parents ligands, crossover finds the largest substructure that the two parent compounds share and generates a child by combining their decoratingmoieties.

Original: Thus, the generation of child ligands are deterministic, and the probability of the generatedligands X childcrossover is
Revised: Thus, the generation of child ligands are deterministic, and the probability of the generatedligands X childcrossover isp crossover ( X child 1 crossover , X child 2 crossover

Original: [Equation]
Revised: |S ( t ) ) =

Original:  X [Equation]
Revised: p crossover ( X parent 1 crossover , X parent 2 crossover |S ( t ) ) = p (1) crossover ( X parent 1 crossover |S ( t ) ) · p (2) crossover ( X parent 2 crossover | X parent 1 crossover , S ( t ) ) .

Original: Mutation Policy Network .
Revised: Mutation Policy Network .

Original: We design two policy networks for two corresponding actions inmutation, as mentioned in Section 3.2.
Revised: We design two policy networks for two corresponding actions inmutation, as mentioned in Section 3.2.

Original: (1) the first action in mutation operation is to select acandidate ligand to be mutated from population S ( t ) .
Revised: (1) the first action in mutation operation is to select acandidate ligand to be mutated from population S ( t ) .

Original: It models the 3D target-ligand complex tolearn if there is improvement space in the current complex.
Revised: It models the 3D target-ligand complex tolearn if there is improvement space in the current complex.

Original: Formally, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as neural architecture, the selection probability of the ligand X parentmutation ∈ S ( t ) isp [Equation] X parentmutation 
Revised: Formally, we obtain a valid probability distribution over all the available ligands based on target-ligand complex as input feature and ENN as neural architecture, the selection probability of the ligand X parentmutation ∈ S ( t ) isp (1) mutation ( X parentmutation |S ( t ) ) =

Original: [Equation][Equation]
Revised: [Equation]

Original:  , where T & X denotes target-ligand complex,h T & X = ENN ( T & X ) represents the ENN’s embedding of target-ligand complex.
Revised: ) (cid:1) , where T & X denotes target-ligand complex,h T & X = ENN ( T & X ) represents the ENN’s embedding of target-ligand complex.

Original: The second action is to select the SMARTS reaction from the reaction set conditioned on the selectedligand in the first action.
Revised: The second action is to select the SMARTS reaction from the reaction set conditioned on the selectedligand in the first action.

Original: Specifically, for each reaction, we generate the new ligand X childmutation , thenobtain the embedding of target, first ligand X parentmutation and the new ligand X childmutation through ENN,concatenate these three embeddings and feed it into an MLP to estimate a scalar as unnormalized probability.
Revised: Specifically, for each reaction, we generate the new ligand X childmutation , thenobtain the embedding of target, first ligand X parentmutation and the new ligand X childmutation through ENN,concatenate these three embeddings and feed it into a MLP to estimate a scalar as unnormalizedprobability.

Original: The unnormalized probabilities for all the reactions are normalized via softmax function, i.e., p (2) mutation ( ξ | X parentmutation , S ( t ) )
Revised: The unnormalized probabilities for all the reactions are normalized via softmax func- tion, i.e., p (2) mutation ( ξ | X parentmutation , S ( t ) )

Original: =
Revised: =

Original: Softmax (cid:8) MLP ( h T ⊕ h X parentmutation ⊕ h X childmutation ]) , · · · , (cid:9) ξ ∈R , where
Revised: Softmax (cid:8) MLP ( h T ⊕ h X parentmutation ⊕ h X childmutation ]) , · · · , (cid:9) ξ ∈R , where

Original: X parentmutation mutated by ξ −−−−−−−→ X childmutation , R is the reaction set.
Revised: X parentmutation mutated by ξ −−−−−−−→ X childmutation , R is the reaction set.

Original: The probability of the generated ligand X childmutationis
Revised: The probability of the generated ligand X childmutationisp mutation ( X childmutation |S ( t ) ) = p (1) mutation ( X parentmutation |S ( t ) ) · p (2) mutation ( [Equation]

Revised: parentmutation

Original: [Equation]
Revised: [Equation]

Revised: S

Revised: [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Original: Policy Gradient .
Revised: Policy Gradient .

Original: We leverage policy gradient to train the target-ligand policy neural network.
Revised: We leverage policy gradient to train the target-ligand policy neural network.

Original: Specifically, we consider maximizing the expected reward as objective via REINFORCE [28],
Revised: Specifically, we consider maximizing the expected reward as objective via REINFORCE [28],

Original: [Equation]
Revised: [Equation]

Original: where p ( X ) is defined in Equation (2) and (3) for crossover and mutation, respectively.
Revised: where p ( X ) is defined in Equation (2) and (3) for crossover and mutation, respectively.

Original: The wholepipeline is illustrated in Figure 1.
Revised: The wholepipeline is illustrated in Figure 1.

Original: To provide a warm start and leverage the structural information,we pretrain the ENNs on 3D target-ligand binding affinity prediction task, where the inputs are thetarget-ligand complexes, and the outputs are their binding affinity.
Revised: To provide a warm start and leverage the structural information,we pretrain the ENNs on 3D target-ligand binding affinity prediction task, where the inputs are thetarget-ligand complexes, and the outputs are their binding affinity.

Original:  Experiment
Revised:  Experiment

Original: In this section, we briefly describe the experimental setup and results.
Revised: In this section, we briefly describe the experimental setup and results.

Original: The Appendix includes moredetails, including software configuration, implementation details, dataset description & processing,hyperparameter tuning, code repository, ablation study, and additional experimental results.
Revised: The Appendix includes moredetails, including software configuration, implementation details, dataset description & processing,hyperparameter tuning, code repository, ablation study, and additional experimental results.

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Docking Simulation .
Revised: Docking Simulation .

Original: We adopt AutoDock Vina [51] to evaluate the binding affinity.
Revised: We adopt AutoDock Vina [51] to evaluate the binding affinity.

Original: The dockingscore estimated by AutoDock Vina is called Vina score and roughly characterizes the free energychanges of binding processes in kcal/mol.
Revised: The dockingscore estimated by AutoDock Vina is called Vina score and roughly characterizes the free energychanges of binding processes in kcal/mol.

Original: Thus lower Vina score means a stronger binding affinitybetween the ligand and target.
Revised: Thus lower Vina score means a stronger binding affinitybetween the ligand and target.

Original: We picked various disease-related proteins, including G-proteincoupling receptors (GPCRs) and kinases from DUD-E [52] and the SARS-CoV-2 main protease [53]as targets.
Revised: We picked various disease-related proteins, including G-proteincoupling receptors (GPCRs) and kinases from DUD-E [52] and the SARS-CoV-2 main protease [53]as targets.

Original: Please see the Appendix for more information.
Revised: Please see the Appendix for more information.

Original: Baselines .
Revised: Baselines .

Original: The baseline methods cover traditional brute-force search methods (Screening), deepgenerative models (JTVAE and Gen3D), genetic algorithm (GA+D, graph-GA, Autogrow 4.0),reinforcement learning methods (MolDQN, RationaleRL, REINVENT, GEGL), and MCMC method(MARS).
Revised: The baseline methods cover traditional brute-force search methods (Screening), deepgenerative models (JTVAE and Gen3D), genetic algorithm (GA+D, graph-GA, Autogrow 4.0),reinforcement learning methods (MolDQN, RationaleRL, REINVENT, GEGL), and MCMC method(MARS).

Original: Gen3D and Autogrow 4.0 are structure-based drug design methods, while others aregeneral-purpose molecular design methods.
Revised: Gen3D and Autogrow 4.0 are structure-based drug design methods, while others aregeneral-purpose molecular design methods.

Original: Although methods explicitly utilizing target structuresare relatively few, we add general-purpose molecular design methods optimizing the same dockingoracle scores as ours, which is a common use case, as baselines [15, 14].
Revised: Although methods explicitly utilizing target structuresare relatively few, we add general-purpose molecular design methods optimizing the same dockingoracle scores as ours, which is a common use case, as baselines [15, 14].

Original: Concretely, Screeningmimics high throughput screening via sampling from ZINC database randomly; JTVAE (Junction
Revised: Concretely, Screeningmimics high throughput screening via sampling from ZINC database randomly; JTVAE (Junction

Original: Tree Variational Auto-Encoder) [20] uses a Bayesian optimization on the latent space to indirectlyoptimize molecules; Gen3D [10] is an auto-regressive generative model that grows 3D structuresatom-wise inside the binding pocket; GA+D
Revised: Tree Variational Auto-Encoder) [20] uses a Bayesian optimization on the latent space to indirectlyoptimize molecules; Gen3D [10] is an auto-regressive generative model that grows 3D structuresatom-wise inside the binding pocket; GA+D

Original: [26] represents molecule as SELFIES string [54] anduses genetic algorithm enhanced by a discriminator neural network; Graph-GA [15] conduct geneticalgorithm on molecular graph representation; Autogrow 4.0 [16] is the state-of-the-art geneticalgorithm in structure-based drug design; MolDQN (Molecule Deep Q-Network) [30] leverages deep
Revised: [26] represents molecule as SELFIES string [54] anduses genetic algorithm enhanced by a discriminator neural network; Graph-GA [15] conduct geneticalgorithm on molecular graph representation; Autogrow 4.0 [16] is the state-of-the-art geneticalgorithm in structure-based drug design; MolDQN (Molecule Deep Q-Network) [30] leverages deep

Original: Q-value learning to grow molecules atom-wisely; RationaleRL [31] uses rationale (e.g., functionalgroups or subgraphs) as the building block and a policy gradient method to guide the training ofgraph neural network-based generator; REINVENT [28] represent molecules as SMILES stringand uses policy gradient based reinforcement learning methods to guide the training of the RNNgenerator; GEGL (genetic expert-guided learning)
Revised: Q-value learning to grow molecules atom-wisely; RationaleRL [31] uses rationale (e.g., functionalgroups or subgraphs) as the building block and a policy gradient method to guide the training ofgraph neural network-based generator; REINVENT [28] represent molecules as SMILES stringand uses policy gradient based reinforcement learning methods to guide the training of the RNNgenerator; GEGL (genetic expert-guided learning)

Original: [32] uses LSTM guided by reinforcement learningto imitate the GA exploration; MARS (Markov Molecule Sampling)
Revised: [32] uses LSTM guided by reinforcement learningto imitate the GA exploration; MARS (Markov Molecule Sampling)

Original: [35] leverages Markov chain
Revised: [35] leverages Markov chain

Original: Monte Carlo sampling (MCMC) with adaptive proposal and annealing scheme to search chemicalspace.
Revised: Monte Carlo sampling (MCMC) with adaptive proposal and annealing scheme to search chemicalspace.

Original: To conduct a fair comparison, we limit the number of oracle calls to 1,000 times for eachmethod.
Revised: To conduct a fair comparison, we limit the number of oracle calls to 1,000 times for eachmethod.

Original: Implementation details of all the baseline methods are available in Appendix.
Revised: Implementation details of all the baseline methods are available in Appendix.

Original: Dataset : we randomly select molecules from ZINC [55] database (around 250 thousands drug-likemolecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, GA+D).
Revised: Dataset : we randomly select molecules from ZINC [55] database (around 250 thousands drug-likemolecules) as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, GA+D).

Original: ZINC alsoserves as the training data for pretraining the model in JTVAE, REINVENT, RationaleRL, etc.
Revised: ZINC alsoserves as the training data for pretraining the model in JTVAE, REINVENT, RationaleRL, etc.

Original: Weadopt CrossDocked2020 [56] dataset that contains around 22 million ligand-protein complexes as [Table] the training data for pretraining the policy neural networks, as mentioned in Section 3.3.
Revised: Weadopt CrossDocked2020 [56] dataset that contains around 22 million ligand-protein complexes asthe training data for pretraining the policy neural networks, as mentioned in Section 3.3.

Original: Moredescriptions are available in Appendix.
Revised: Moredescriptions are available in Appendix.

Revised: [Table]

Original: Metrics .
Revised: Metrics .

Original: The selection of evaluation metrics follows recent works in molecule optimization [20, 26,31, 35] and structure-based drug design [16, 10, 14].
Revised: The selection of evaluation metrics follows recent works in molecule optimization [20, 26,31 , 35] and structured based drug design [16, 10, 14].

Original: For each method, we select top-100 moleculeswith the best docking scores for evaluation and consider the following metrics: TOP-1/10/100(average docking score of top-1/10/100 molecules): docking score directly measures the bindingaffinity between the ligand and target and is the most informative metric in structure-based drugdesign; Novelty (Nov) (% of the generated molecules that are not in training set); Diversity (Div)(average pairwise Tanimoto distance between the Morgan fingerprints); We also evaluate some simplepharmaceutical properties, including quantitative drug-likeness ( QED ) and synthetic accessibility( SA ).
Revised: For each method, we select top-100 moleculeswith the best docking scores for evaluation and consider the following metrics: TOP-1/10/100(average docking score of top-1/10/100 molecules): docking score directly measures the bindingaffinity between the ligand and target and is the most informative metric in structure-based drugdesign; Novelty (Nov) (% of the generated molecules that are not in training set); Diversity (Div)(average pairwise Tanimoto distance between the Morgan fingerprints); We also evaluate some simplepharmaceutical properties, including quantitative drug-likeness ( QED ) and synthetic accessibility( SA ).

Original: QED score indicates drug-likeliness ranging from 0 to 1 (higher the better).
Revised: QED score indicates drug-likeliness ranging from 0 to 1 (higher the better).

Original: SA score rangesfrom 1 to 10 (lower the better).
Revised: SA score rangesfrom 1 to 10 (lower the better).

Original: 4.2 Results
Revised: 4.2 Results

Original: Stronger Optimization Performance .
Revised: Stronger Optimization Performance .

Original: We summarized the main results of the structure-baseddrug design in Table 1.
Revised: We summarized the main results of the structure-baseddrug design in Table 1.

Original: We evaluate all the methods on all targets and report each metric’s meanand standard deviations across all targets.
Revised: We evaluate all the methods on all targets and report each metric’s meanand standard deviations across all targets.

Original: Our result shows RGA achieves the best performance in
Revised: Our result shows RGA achieves the best performance in

Original: TOP-100/10/1 scores among all methods we compared.
Revised: TOP-100/10/1 scores among all methods we compared.

Original: Compared to Autogrow 4.0, RGA ’s betterperformance in docking score demonstrates that the policy networks contribute positively to thechemical space navigation and eventually help discover more potent binding molecules.
Revised: Compared to Autogrow 4.0, RGA ’s betterperformance in docking score demonstrates that the policy networks contribute positively to thechemical space navigation and eventually help discover more potent binding molecules.

Original: On the otherhand, including longer-range navigation steps enabled by crossover leads to superior performancethan other RL methods (REINVENT, MolDQN, GEGL and RationaleRL) that only focus on localmodifications.
Revised: On the otherhand, including longer-range navigation steps enabled by crossover leads to superior performancethan other RL methods (REINVENT, MolDQN, GEGL and RationaleRL) that only focus on localmodifications.

Original: In addition, we also observed competitive structure quality measured by QED ( > 0 . 7 )and SA_Score ( < 2 . 5 ) in Autogrow 4.0 and RGA without involving them as optimization objectives,thanks to the mutation steps originating from chemical reactions.
Revised: In addition, we also observed competitive structure quality measured by QED ( > 0 . 7 )and SA_Score ( < 2 . 5 ) in Autogrow 4.0 and RGA without involving them as optimization objectives,thanks to the mutation steps originating from chemical reactions.

Original: We visualize two designed ligandswith optimal affinity for closer inspection in Figure 2(a) and 2(b), and find both ligands bind tightlywith the targets.
Revised: We visualize two designed ligandswith optimal affinity for closer inspection in Figure 2(a) and 2(b), and find both ligands bind tightlywith the targets.

Original: Suppressed Random-Walk Behavior .
Revised: Suppressed Random-Walk Behavior .

Original: Especially in SBDD, when the oracle functions are expensive molecular simulations, robustness to random seeds is essential for improving the worst-caseperformance of algorithms.
Revised: Especially in SBDD, when the oracle functions are expensive molecular simulations, robustness to random seeds is essential for improving the worst-caseperformance of algorithms.

Original: One of the major issues in traditional GAs is that they have a significant variance between multiple independent runs as they randomly select parents for crossover andmutation types.
Revised: One of the major issues in traditional GAs is that they have a significant variance between multiple independent runs as they randomly select parents for crossover andmutation types.

Original: To examine this behavior, we run five independent runs for RGA , Autogrow 4.0 andgraph-GA
Revised: To examine this behavior, we run five independent runs for RGA , Autogrow 4.0 andgraph-GA

Original: (three best baselines, all are GA methods) on all targets and plot the standard deviationsbetween runs in Figure 2(c) and 2(d).
Revised: (three best baselines, all are GA methods) on all targets and plot the standard deviationsbetween runs in Figure 2(c) and 2(d).

Original: With policy networks guiding the action steps, we observed thatthe random-walk behavior in Autogrow 4.0 was suppressed in RGA , indicated by the smaller variance.
Revised: With policy networks guiding the action steps, we observed thatthe random-walk behavior in Autogrow 4.0 was suppressed in RGA , indicated by the smaller variance.

Original: Especially in the later learning phase (after 500 oracle calls), the policy networks are fine-tuned andguide the search more intelligently.
Revised: Especially in the later learning phase (after 500 oracle calls), the policy networks are fine-tuned andguide the search more intelligently.

Original: This advantage leads to improved worst-case performance and ahigher probability of successfully identifying bioactive drug candidates with constrained resources.[Figure]
Revised: This advantage leads to improved worst-case performance and ahigher probability of successfully identifying bioactive drug candidates with constrained resources.

Original: Knowledge Transfer Between Protein Targets .
Revised: Knowledge Transfer Between Protein Targets .

Original: To verify if RGA benefited from learning the sharedphysics of ligand-target interaction, we conducted an ablation study whose results are in the lasttwo rows of Table 1.
Revised: To verify if RGA benefited from learning the sharedphysics of ligand-target interaction, we conducted an ablation study whose results are in the lasttwo rows of Table 1.

Original: Specifically, we compare RGA with two variants: (1) RGA -pretrain that does notpretrain the policy network with all native complex structures in the CrossDocked2020; (2) RGA -KT(knowledge transfer) that fine-tune the networks with data of individual target independently.
Revised: Specifically, we compare RGA with two variants: (1) RGA -pretrain that does notpretrain the policy network with all native complex structures in the CrossDocked2020; (2) RGA -KT [Figure] (knowledge transfer) that fine-tune the networks with data of individual target independently.

Original: Wefind that both strategies positively contribute to RGA on TOP-100/10/1 docking score.
Revised: Wefind that both strategies positively contribute to RGA on TOP-100/10/1 docking score.

Original: These resultsdemonstrate the policy networks successfully learn the shared physics of ligand-target interactionsand leverage the knowledge to improve their performance.
Revised: These resultsdemonstrate the policy networks successfully learn the shared physics of ligand-target interactionsand leverage the knowledge to improve their performance.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose Reinforced Genetic Algorithm ( RGA ) to tackle the structure-based drugdesign problem.
Revised: In this paper, we propose Reinforced Genetic Algorithm ( RGA ) to tackle the structure-based drugdesign problem.

Original: RGA reformulate the evolutionary process in genetic algorithms as a Markov decisionprocess called evolutionary Markov decision process (EMDP) so that the searching processes couldbenefit from trained neural models.
Revised: RGA reformulate the evolutionary process in genetic algorithms as a Markov decisionprocess called evolutionary Markov decision process (EMDP) so that the searching processes couldbenefit from trained neural models.

Original: Specifically, we train policy networks to choose the parents tocrossover and mutate instead of randomly sampling them.
Revised: Specifically, we train policy networks to choose the parents tocrossover and mutate instead of randomly sampling them.

Original: Further, we also leverage the commonphysics of the ligand-target interaction and adopt a knowledge-transfer strategy that uses data fromother targets to train the networks.
Revised: Further, we also leverage the commonphysics of the ligand-target interaction and adopt a knowledge-transfer strategy that uses data fromother targets to train the networks.

Original: Through empirical study, we show that RGA has strong and robustoptimization performance, consistently outperforming baseline methods in terms of docking score.
Revised: Through empirical study, we show that RGA has strong and robustoptimization performance, consistently outperforming baseline methods in terms of docking score.

Original: Though we adopted mutations originating from chemical reactions and the structural quality metricsseem good, we need to emphasize that the designed molecules from RGA do not guarantee synthesizability [48], as the crossover operations may break inheriting synthesizability.
Revised: Though we adopted mutations originating from chemical reactions and the structural quality metricsseem good, we need to emphasize that the designed molecules from RGA do not guarantee synthesizability [48], as the crossover operations may break inheriting synthesizability.

Original: Directly working onsynthetic pathways could solve the problem [27, 57], but the extension is not trivial.
Revised: Directly working onsynthetic pathways could solve the problem [27, 57], but the extension is not trivial.

Original: As for futuredirection, we expect to theoretically analyze the EMDP formulation and the performance of RGA .
Revised: As for futuredirection, we expect to theoretically analyze the EMDP formulation and the performance of RGA .

Original: Wealso expect to generalize RGA to other combinatorial optimization scenarios, such as symbolic lawsdiscovery [58], quantum circuits design [59]), etc.
Revised: Wealso expect to generalize RGA to other combinatorial optimization scenarios, such as symbolic lawsdiscovery [58], quantum circuits design [59]), etc.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] See the
Revised: [Yes] See the

Original: supplementary materials.
Revised: supplementary materials.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,
Revised: g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See the supplementary
Revised: [Yes] See the supplementary

Original: materials.
Revised: materials.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See the supplementary materials.
Revised: [Yes] See the supplementary materials.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: See the supplementary materials.
Revised: See the supplementary materials.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: ) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A] Contents
Revised: ) Did you include the estimated hourly wage paid to participants and the total amount

Original: 1 Introduction

Original: 2 Related Works

Original: 3 Method 33.

Original: Evolutionary Process . .

Original: Evolutionary Markov Decision Process . .

Original: Target-Ligand Policy Network . .

Original: 54 Experiment 74.

Original: Experimental Setup . .

Original: Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: [-85 Conclusion 
-]
Revised: [-85 Conclusion 
-]

Original: [-A Mathematical Notation 
-]
Revised: [-A Mathematical Notation 
-]

Original: [-B Illustration of Genetic Algorithm 
-]
Revised: [-B Illustration of Genetic Algorithm 
-]

Original: [-C Baseline Setup 
-]
Revised: [-C Baseline Setup 
-]

Original: [-D Additional Experimental Setup 
-]
Revised: [-D Additional Experimental Setup 
-]

Original: D.1 Docking Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 Dataset . .

Original: D.3 Evaluation metrics . .

Original: [-E Implementation Details 
-]
Revised: [-E Implementation Details 
-]

Original: E.1 Software/Hardware Configuration . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2 Hyperparameter Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.3 Code Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Additional Experiment

Original: F.

Original: Efficiency Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.

Original: Pretraining Equivariant neural network . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.

Original: Additional Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F. Scalability . . . . F.
Revised: spent on participant compensation?

Original: Significance Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F.

Original: Example of the generated ligand . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G Additional Discussion on Related Work

Original: A Mathematical Notation

Original: For ease of exposition, we list the mathematical notations in Table 2.

Original: All the mathematical notations are divided into three parts: (1) notation for genetic algorithm (Section 3.1); (2) notation forequivariance neural networks (ENN) [18] (Section 3.3); (3) notations for policy network (Section 3.3).

Original: [Table]

Original: that embeds the i -th node

Original: Z ( l ) =

Original: { z ( l ) i } Ni =

Original: Coordinate embeddings at the l -th layer w ( l ) ij message vector for the edge from node i to node j at l -th layer v ( l ) i message vector

Original: [Table]

Original: node i

Original: [Table]

Original: th layer z ( l ) i the position embedding for node i at l -th layer h ( l ) i the node embedding for node i at l -th layer h

Original: Y = ENN ( Y )

Original: ENN representation of the 3D graph Y (Equation 1) p (1)

Original: [Table]

Original: ( X parent 1 crossover |S ( t ) )

Original: [Table]

Original: crossover p

Original: [Table]

Original: [Table]

Original: | X

Original: [Table]

Original: crossover , S

Original: [Table]

Original: t ) )

Original: [Table]

Original: p

Original: [Table]

Original: S

Original: [Table]

Original: S

Original: [Table]

Original: p

Original: [Table]

Original: [Table]

Original: t

Original: [Table]

Original: [Table]

Original: B Illustration of Genetic Algorithm

Original: Figure 3 provides two examples to illustrate crossover and mutation operations in genetic algorithmdescribed in Section 3.1.

Original: Crossover , also called recombination, combines the structure of two parents to generate new children.

Original: Following Autogrow 4.0 [16], as shown in Figure 3(a), we select two parents from the last generationand search for the largest common substructure shared between them.

Original: Then we generate two childrenby randomly switching their decorating moieties, i.e., the side chains attached to the commonsubstructure.

Original: Mutation operator performs an in silico chemical reaction to generate an altered child compound(i.e.

Original: , product in the chemical reaction) derived from a parent (reactant in the chemical reaction), asshown in Figure 3(b).

Original: The chemical reaction here contains two reactants, one is parent molecule,another is from reaction set.

Original: The reaction set R is generated via merging two public reaction libraries:(1)

Original: the AutoClickChemRxn set (36 reactions) [46] and (2)

Original: RobustRxn set (58 reactions [49]).

Original: Eachreaction ξ in reaction set R contains a SMARTS string based reaction template and a reactant.

Original: It uses

Original: SMARTS reaction template, together with RDKit [60], to perform chemical mutations efficiently.

Original: The process is written as X parentmutation mutated by ξ −−−−−−−→ X childmutation , where ξ is selected reaction (with a reactiontemplate and another reactant).

Original: The ligand to be mutated and the reaction used for mutation are bothrandomly selected from previous generation and reaction set R , respectively.

Original: Compared with themutation operator in conventional GA that randomly flipping an arbitrary bit, the reaction-basedmutation enhance synthesizability of the generated molecules [16].

Original: Mutation operator performs an insilico chemical reaction to generate an altered child compound (i.e., product in the chemical reaction)derived from a parent (reactant in the chemical reaction).

Original: The chemical reaction here contains tworeactants, one is parent molecule, another is from reaction set.

Original: [Figure]

Original: C Baseline Setup

Original: In this section, we describe the experimental setting for baseline methods.

Original: Most of the settings followthe original papers.

Original: • GA+D

Original: (genetic algorithm enhanced by discriminator neural network) [26] utilizes SELFIESstring as the representation of molecules, thus guaranteeing the 100% chemical validity ofthe generated molecules.

Original: Following their original paper, the discriminator neural network isa two-layer fully connected neural network with ReLU activation and sigmoid output layer.

Original: The hidden size is 100.

Original: the size of the output layer is 1.

Original: The input feature of discriminator neural network is a vector of chemical and geometrical properties characterizing themolecules.

Original: The population size is set to 300.

Original: Maximal generation number is set to 1000.The patience is set to 5.

Original: When the property does not improve when the patience exhausts,the process early stops.

Original: We used Adam optimizer with 1e-3 as the initial learning rate.

Original: beta( β ) is the weight of discriminator neural network’s score in fitness evaluation, which is usedto select most promising molecules in each generation.

Original: We set β = 1 .•

Original: Graph-GA [15] uses molecular graph to represent molecules and uses crossover andmutation operations to edit the molecular graph.

Original: After tuning, the size of population is setto 120.

Original: The size of offspring is set to 70.

Original: The mutation rate is set to 0.067.

Original: Graph-GAdo not have learnable parameters and is easy to implement.

Original: We use the implementation inGuacaMol [13].• MolDQN (Molecule Deep Q-Networks)

Original: [30] uses molecular graph to represent molecules,formulate the molecule optimization process as a Markov Decision Process (MDP) and

Original: utilize Deep Q-value learning to optimize it.

Original: It grows molecular graph atom-wise, that is,

Original: [Table]

Original: in each episode, it adds one atom to the partially generated molecular graph.

Original: The reward

Original: [Table]

Original: function is the negative value of the Vina score.

Original: Following the original paper, maximalstep in each episode is 40.

Original: Each step calls oracle once.

Original: The discount factor is 0.9.

Original: DeepQ-network is a multilayer perceptron (MLP) whose hidden dimensions are 1024, 512, 128,32, respectively.

Original: The model size is 6.4 M. The input of the Q-network is the concatenationof the molecule feature (2048-bit Morgan fingerprint, with a radius of 3) and the number ofleft steps.

Original: Adam is used as an optimizer with 1e-4 as the initial learning rate.

Original: Only ringswith a size of 5 and 6 are allowed.

Original: It leverages ϵ -greedy together with randomized value

Original: [Table]

Original: functions (bootstrapped-DQN) as an exploration policy, ϵ is annealed from 1 to 0.01 in apiecewise linear way.• RationaleRL

Original: The architecture of the generator is a message-passing network (MPN)followed by MLPs applied in breadth-first order.

Original: The generator is pre-trained on generalmolecules combined with an encoder and then fine-tuned to maximize the reward functionusing policy gradient.

Original: The encoder and decoder MPNs both have hidden dimensions of 400.The dimension of the latent variable is 50.

Original: Adam optimizer is used on both pre-training andfine-tuning with initial learning rates of 1e-3, 5e-4, respectively.

Original: The annealing rate is 0.9.We pre-trained the model with 20 epochs.

Original: [Table]

Original: • MARS [35] leverage Markov chain Monte Carlo sampling (MCMC) on molecules with

Original: [Table]

Original: an annealing scheme and an adaptive proposal.

Original: The proposal is parameterized by a graph

Original: [Table]

Original: neural network, which is trained on MCMC samples.

Original: We follow most of the settings in the

Original: [Table]

Original: original paper.

Original: The message passing network has six layers, where the node embedding

Original: [Table]

Original: size is set to 64.

Original: Adam is used as an optimizer with 3e-4 initial learning rate.

Original: To generate abasic unit, top-1000 frequent fragments are drawn from ZINC database [55] by enumeratingsingle bonds to break.

Original: During the annealing process, the temperature T = 0 .

Original: ⌊ t/ 5 ⌋ wouldgradually decrease to 0.•

Original: Autogrow 4.0 [16] is the base model for RGA and have been briefly described in Section 3.1.The setup of Autogrow is the same as RGA for fair comparison, the only difference is thatRGA use policy network to guide the selection of ligands and reaction for crossover andmutation while Autogrow randomly selects them.

Original: The reaction set R is generated viamerging two public reaction libraries: (1) the AutoClickChemRxn set (36 reactions)

Original: [46]and (2)

Original: RobustRxn set (58 reactions [49]).

Original: In each generation, It generates 200 offspring(100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vinascores) ones for the next generation.•

Original: Screening exhaustively searches the ZINC database [55] within oracle budget.

Original: It is thetraditional high-throughput screening approach.• REINVENT [28] is a reinforcement learning approach, represent molecule as SMILESstring and uses recurrent neural network to model SMILES string.

Original: It pretrains a priormodel using molecules on ZINC and finetune the model using the reward function.

Original: It usesREINFORCE to maximize the expected reward function.

Original: The learning rate is set to 0.0005;the batch size is set to 64; The hyperparameter σ weighs the pretrained prior model and thereward function, and is set to 60.

Original: The model size is 16.3M.• JTVAE [20] build a junction tree to represent molecule via using substructure (either ringor atom) to represent molecule.

Original: It uses both molecular graph-leven and junction tree-levelencoder and decoder.

Original: The VAE model is pretrained on ZINC databases.

Original: Then BayesianOptimization is used to optimize the docking score on the continuous latent space.

Original: Weuse “botorch”, the python’s Bayesian optimization package, to implement the Bayesianoptimization process.

Original: It has 703 substructures in vocabulary, extracted from ZINC.

Original: Thehidden size is 450.

Original: The latent size of VAE is set to 56.

Original: The model size is 21.8 M.• Gen3D [10] uses 3D deep generative models and grow the molecule via adding atomsauto-regressively.

Original: It train a universal model for all the targets.

Original: The number of messagepassing layers in context encoder is 6, and the hidden dimension is 256.

Original: We train the modelusing the Adam optimizer at learning rate 0.0001.

Original: The model size is 17.4 M.• GEGL (Genetic Expert Guided Learning)

Original: [32] uses LSTM (guided by RL agent) to imitateGA process, however, it is unable to inherit the GA’s flexible assembling manner due to the

Original: auto-regressive essence of LSTM.

Original: It use Adam as optimizer with initial learning rate 1e-3.

Original: The batch size during sampling is 512, the batch size during optimization is 256.

Original: In GA,mutation rate is 0.01.

Original: The similarity threshold is 0.4, which constrain the similarity betweenthe original molecule and the edited molecule.

Original: The maximal SMILES length is set to 120.

Original: D Additional Experimental SetupD.1 Docking Simulation

Original: Molecular docking is a computational method which predicts the preferred orientation of one moleculeto a second when a ligand and a target are bound to each other to form a stable complex.

Original: Knowledgeof the preferred orientation in turn may be used to predict the strength of association or bindingaffinity between two molecules using, for example, scoring functions.

Original: We adopt AutoDock Vina [51]to evaluate the binding affinity.

Original: The docking score estimated by AutoDock Vina is called Vina scoreand roughly characterizes the free energy changes of binding processes in kcal/mol.

Original: Vina score isusually smaller than 0 and lower Vina score means a stronger binding affinity between the ligand andtarget.

Original: We leverage the negative value of the docking score as reward function (Equation 4).

Original: D.

Original: 2 Dataset

Original: In this paper, we use ZINC [55] database and CrossDocked2020 [56] dataset.

Original: ZINC is a free databaseof 250 thousands commercially-available drug-like chemical compounds for virtual screening [55].

Original: We randomly select molecules from ZINC [55] database (around 250 thousands drug-like molecules)as 0-th generation of the genetic algorithms ( RGA , Autogrow 4.0, graph-GA GA+D).

Original: Other baselinemethods also use ZINC to either pretrain the models, e.g., JTVAE, REINVENT, RationaleRL orprovide searching database, e.g., screening.

Original: We adopt CrossDocked2020 [56] dataset that containsaround 22 million ligand-protein complexes as the training data for pretraining the policy neuralnetworks, as mentioned in Section 3.3.

Original: Regarding the target proteins, we picked various disease-related proteins, including G-protein coupling receptors (GPCRs) and kinases from DUD-E [52] and the SARS-CoV-2 main protease [53] astargets.

Original: for all the selected target protein, the binding pocket size for all the targets are set to (15.0,15.0, 15.0).

Original: The units of coordinate are Angstrom ˚ A ( 10 − 10 m).

Original: Detailed descriptions of these targetsare available at https://www.rcsb.org/ .

Original: D.3 Evaluation metrics

Original: We leverage the following evaluation metrics to measure the optimization performance:

Original: • Novelty is the fraction of the generated molecules that do not appear in the training set.

Original: • Diversity of generated molecules is defined as the average pairwise Tanimoto distance

Original: between the Morgan fingerprints [29, 31, 35].

Original: [Equation]

Original: where M is the set of generated molecules that we want to evaluate.

Original: sim ( m 1 , m 2 ) is

Original: the Tanimoto similarity between molecule m 1 and m 2 ,

Original: where (Tanimoto) Similarity mea-

Original: sures the similarity between the input molecule and generated molecules.

Original: It is defined as

Original: [Equation]

Original: is the binary Morgan fingerprint vector for the molecule

Original: [Equation]

Original: X .

Original: In this paper, it is a 2048-bit binary vector.• QED represents a quantitative estimate of drug-likeness.

Original: QED score ranges from 0 to 1.

Original: Itcan be evaluated by the RDKit package ( https://www.rdkit.org/ ).

Original: • SA (Synthetic Accessibility) score measures how hard it is to synthesize a given molecule,based on a combination of the molecule’s fragments contributions [61].

Original: It is evaluated via

Original: RDKit [60].

Original: The raw SA score ranges from 1 to 10.

Original: A higher SA score means the moleculeis hard to be synthesized and is not desirable.

Original: • Run Time .

Original: Unlike optimizing some simple oracles such as QED and LogP scores, the

Original: docking simulation need to search 3D molecular conformation docked in the target, which is computationally expensive.

Original: Thus run time is an important metric to measure the efficiencyof the methods.

Original: E Implementation DetailsE.1 Software/Hardware Configuration

Original: We implemented RGA using Pytorch 1.10.2, Python 3.7, RDKit v2020.09.1.0 on an Intel Xeon

Original: E5-2690 machine with 256G RAM and NVIDIA Pascal Titan X GPUs.

Original: E.2 Hyperparameter Setup

Original: The neural architectures of policy networks are E(3)-equivariant neural network (ENN) [18].

Original: Thevocabulary set V = { C, N, O, S, H, other } .

Original: In ENN, the number of layers is set to 3, i.e., L = 3 ;the hidden dimension is set to 100, i.e., d = 100 .

Original: In Equation 1, MLP e ( · ) , MLP x ( · ) , MLP h ( · ) aretwo-layer MLP in ENN with Swish activation [50] in hidden layer.

Original: Summation function is usedas aggregation function to aggregate the last-layer’s node embedding into graph-level embedding.

Original: All the atoms that within the binding site are used as the input of ENN.

Original: REINFORCE is used toimplement policy gradient [62, 28].

Original: Adam is utilized as optimizer with learning rate 0 .

Original: 001 for bothcrossover and mutation policy networks.

Original: The reaction set R is generated via merging two publicreaction libraries: (1) the AutoClickChemRxn set (36 reactions) [46] and (2) RobustRxn set (58reactions [49]).

Original: We use RDKit [60] to perform in silico chemical reaction based on SMARTS reactiontemplate, then we have |R| = 36 + 58 = 94 .

Original: In each generation, we generate up to 200 offspring(100 from crossover and 100 from mutation) and keep 50 most promising (with lowest Vina scores)ones for the next generation, i.e., K = 50 .

Original: E.3 Code Repository

Original: The code repository is uploaded in supplementary material for reproducibility.

Original: F Additional ExperimentF.1 Efficiency Study

Original: As mentioned before, unlike optimizing some simple oracles such as QED and LogP scores, thedocking simulation need to search 3D molecular conformation docked in the target, which is timeconsuming.

Original: Thus run time is an important measurement to evaluate the efficiency of the methods.

Original: We report the bar of run time over different targets for all the compared methods in Figure 4.

Original: Therun times varies greatly over different methods.

Original: Thus, for ease of visualization, we divided all themethods into two groups.

Original: One is slow group, containing 5 methods: screening, GEGL, REINVENT,

Original: RationaleRL and JTVAE, where all the methods take more than 10 hours.

Original: Another is fast group ( < 10hours), containing 7 methods, MARS, MolDQN, Gen3D, GA+D, GraphGA, Autogrow, and RGA.

Original: We find that both Autogrow and RGA are efficient compared with other methods.

Original: This attributes tothe unique design of genetic algorithm (both crossover and mutation operations) and the usage offilter after GA operators, as described in Section 3.1.

Original: RGA is only slightly slower than Autogrowbecause it requires additional computation to pretrain/train the policy neural networks.

Original: F.2 Pretraining Equivariant neural network

Original: Pretraining equivariant neural network is a crucial step to RGA .

Original: We adopt CrossDocked2020 [56]dataset that contains around 22 million ligand-protein complexes as the training data for pretraining thepolicy neural networks, as mentioned in Section 3.3.

Original: We split the whole dataset into training/validationdataset with ratio of 9:1.

Original: Each data point is a target-ligand complex and the binding affinity (scalar).

Original: We report the validation loss of ENN on target-ligand binding affinity prediction task.

Original: The validationloss function is root mean square error (RMSE).

Original: We find the learning process converges rapidly whenpassing 150K data points (within an epoch) in terms of validation MSE loss.

Original: [Figure]

Original: F.3 Additional Ablation Study

Original: [Figure]

Original: To further understand our model and GA process, we conduct an ablation study to investigate theimpact of each component/strategy to optimization performance.

Original: Specifically, we consider thefollowing four variants of RGA.

Original: RGA -pretrain is a variant of RGA that does not pretrain the policyneural network.

Original: RGA -KT (Knowledge Transfer) is a variant of RGA that does not training policyneural network on different target proteins, i.e., optimizing ligand for one target at a time.

Original: RGA -MU(mutation) is a variant of RGA that does not involve mutation operation in GA.

Original: That is, all the ligandsare generated via crossover operator.

Original: Correspondingly, RGA -CO (crossover) is a variant that doesnot use crossover operation in GA, which means no mutation operator.

Original: The results are reportedin Table 3.

Original: We find that removing either component/strategy will cause a drop in optimizationperformance (i.e., increase in TOP-100/10/1 scores).

Original: Both crossover and mutation are critical tothe optimization performance.

Original: Also, both pretraining the policy networks and knowledge transferbetween different target have positive contribution to the performance.

Original: The ablation study furtherlyvalidates the effectiveness of the proposed RGA method.

Original: affinity prediction.

Original: We plot the root-mean-square error (RMSE) loss as a function of number of passed data.

Original: We use early stop strategy to terminate the learning process earlier when the validation loss would not decrease to save computational resource and avoid overfitting.

Original: We found the learning process would converges when passing 150K data samples (within an epoch), RMSE loss decreases from more than 8 to less than 1.

Original: [Table]

Original: F.4 Scalability

Original: As mentioned before, the population size in RGA is K .

Original: Then we analyze the computational complexitywithin each generation.

Original: There are two operations, including crossover and mutation operations, asdescribed in Section 3.3.

Original: For crossover, the first step is to select the first parent molecule, we need to evaluate the probabilityover all the molecules in current population, whose complexity is O ( K ) .

Original: The second step is to selectthe second parent molecule based on the first parent molecule, we need to evaluate the probabilityover all the remaining molecules in current population, as shown in Equation (2), whose complexityis O

Original: ( K − 1) .

Original: The complexity of crossover operation is O ( K ) .

Original: On the other hand, for mutation operation, the first step is to select the parent molecule (onlyone parent) via evaluating the probability over all the molecules in the current population, i.e.,p (1) mutation ( X parentmutation |S ( t ) ) , whose complexity is O ( K ) .

Original: The second step is to select a mutated molecules(generated by chemical reaction), as shown in Equation (3).

Original: The complexity is O ( |R| ) , where R isthe reaction set (see Table 2).

Original: The complexity of mutation operation is O ( |R| + K ) .

Original: To summarize, the complexity of RGA is O

Original: ( K + |R| ) .

Original: That is, RGA scales linearly in population size

Original: K and the size of the chemical reaction set |R| .

Original: As mentioned in Section E.2, |R| = 94 , K = 50 .

Original: Thus, RGA owns desired scalability and is not computational expensive.

Original: [Table]

Original: F.5 Significance Studies

Original: [Table]

Original: In this section, we present the significance studies.

Original: Specifically, we conduct the hypothesis testingto show the statistical significance of our method over the other GA methods.

Original: We compare thesignificance of the improvement over other GA methods (including AutoGrow4, Graph-GA, GA+D,and GEGL) via running 5 independent trials with different random seeds and thep-value .

Original: We consider the top-100 and top-10 score as the major metrics, which are the most importantmetrics for optimization performance.

Original: We compare RGA versus AutoGrow 4, Graph-GA and GA+D.

Original: The results (p-value) are shown in Table 4.

Original: We find almost all the p-values are less than 0.05 (exceptone value), which indicates that the improvements of RGA over other GA methods are statisticallysignificant.

Original: F.6 Example of the generated ligand

Original: This section shows some examples of the generated ligands with desirable binding affinity for varioustarget proteins in Figure 6, 7, 8 and 9, respectively.

Original: We observe that the generated ligands bind tightlywith the target proteins.

Original: G Additional Discussion on Related Work

Original: Methodology .

Original: The molecule generations methods can be divided into two categories.

Original: The firstone is deep generative models (DGMs), which leverage the continuous representation to estimatethe data distribution using various kinds of deep neural networks, including variational autoencoder(VAE)

Original: [19, 20], generative adversarial network (GAN) [21, 22], normalizing flow model [23, 63, 24],energy based model [25] and diffusion model [64], etc.

Original: The second one is combinatorial optimizationmethods, which directly search the discrete chemical space, including genetic algorithm (GA) [15, 26,27], reinforcement learning approaches (RL) [28, 29, 30, 31], Bayesian Optimization (BO) [33, 65],

Original: Monte Carlo Tree Search (MCTS) [66, 67, 31] and Markov Chain Monte Carlo (MCMC) [34, 35, 36].

Original: Deep generative models (DGMs) usually require a large amount of data to fit, which impedes theirusage in low data regime.

Original: In contrast, combinatorial optimization methods require less training data,while the trade-off is the need to call the optimization oracles during the exploration in the chemicalspace [30, 37, 48, 27, 13].

Original: Among all the machine learning methods, Genetic algorithm (GA) exhibits superior performance insome standard benchmarks [13, 14].

Original: The key reason is GA’s global assembling strategy.

Original: Specifically,in each generation (iteration), GA maintains a population of molecule candidates (a.k.a.

Original: parents), andconducts the crossover operation between two (random-selected) parent candidates, which enablesrelatively large exchanges on molecular sub-graph between molecular graphs.

Original: However, GA isleveraging random-walk based mutation and crossover operations [15, 16] and is essentially based onbrute-force trial and error.

Original: On the other hand, Reinforcement learning (RL) methods are good at navigating the discrete space viaprioritizing the promising searching branches and circumventing brute-force search.

Original: The current RLbased methods [28, 29, 30, 31] slightly left behind other state-of-the-art combinatorial optimizationmethods [37, 14].

Original: The main reason is that current RL based molecule optimization approaches arebased on auto-regressive assembling strategy, i.e., growing molecules iteratively via adding a basic

Original: [Figure]

Original: building block one time, where the building block can be either a token in SMILES representation [28]or a substructure in molecular graph representation [29, 30, 31].

Original: Such assembling strategy areessentially local search methods, which hinders the algorithm’s ability to overcome the roughoptimization landscape (or energy barrier) and is easy to be stuck in the local optimum [68, 69].

Original: Discussion .

Original: Among all the machine learning methods, molecular graph level genetic algorithm (GA)exhibits state-of-the-art performance in some standard molecule optimization benchmarks [13, 14].

Original: The key reason is GA’s assembling manner.

Original: Specifically, in each generation (iteration), GA maintainsa population of possible candidates (a.k.a. parents), and conducts the crossover operation betweentwo candidates to generate new offspring, which enables thorough exploration to the chemical space.

Original: However, there is still improvement space for GA.

Original: GA are leveraging random-walk based mutationand crossover operations [15] and suffers from brute-force trial and error strategy.

Original: [Figure]

Original: On the other hand, reinforcement learning approaches are good at navigating the discrete space viaprioritizing the promising decisions that are worth investigating, for example, AlphaGo successfullyapplied RL to defeat a professional human Go player [70].

Original: However, the current RL based drugdesign methods [28, 29, 30] slightly left behind other state-of-the-art combinatorial optimizationmethods.

Original: The main reason lies at the inferior assembling strategy, which grows molecule in anauto-regressive fashion.

Original: It is hard for this kind of local search strategy to overcome the barrier of theobjective, so it is easy to be trapped into the local optimum.

Original: [Figure]
Revised: [N/A]

Original: Deep learning methods can also enhance genetic algorithm.

Original: Due to the random selection used ingenetic algorithm, it is challenging to apply deep learning methods in the generation of new candidates(molecules in this paper).

Original: Deep learning can be used to compose fitness evaluation to select theoffspring.

Original: For example, GA+D

Original: [26] leverages deep neural network as a discriminator to measure thedrug’s proximity to the training data, which is incorporated as a scorer in fitness evaluation.

Original: [71] traina deep neural network-based property predictor and leverage it to enhance the evolutionary algorithm.

Original: In this paper, we attempt to enhance genetic algorithm using reinforcement learning technique.

Original: Specifically, we propose Reinforced Genetic Algorithm ( RGA ), which inherits the assembling mannerfrom genetic algorithm and use reinforcement learning to guide the search over the chemical space.[32] also combine RL and GA, which uses LSTM (guided by RL agent) to imitate GA process,however, it is unable to inherit the GA’s flexible assembling manner due to the auto-regressive essenceof LSTM.

Original: [Figure]