Original: P RACTICAL M ASSIVELY P ARALLEL M ONTE -C ARLO T REE S EARCH A PPLIED TO M OLECULAR D ESIGN
Revised: P RACTICAL M ASSIVELY P ARALLEL M ONTE -C ARLO T REE S EARCH A PPLIED TO M OLECULAR D ESIGN

Original: A BSTRACT
Revised: A BSTRACT

Original: It is common practice to use large computational resources to train neural networks, known from many examples, such as reinforcement learning applications.
Revised: It is common practice to use large computational resources to train neural networks, known from many examples, such as reinforcement learning applications.

Original: However, while massively parallel computing is often used for training models, it is rarely used to search solutions for combinatorial optimization problems.
Revised: However, while massively parallel computing is often used for training models, it is rarely used to search solutions for combinatorial optimization problems.

Original: This paper proposes a novel massively parallel Monte-Carlo Tree Search (MP-MCTS) algorithm that works efﬁciently for a 1,000 worker scale on a distributed memory environment using multiple compute nodes and applies it to molecular design.
Revised: This paper proposes a novel massively parallel Monte-Carlo Tree Search (MP-MCTS) algorithm that works efﬁciently for a 1,000 worker scale on a distributed memory environment using multiple compute nodes and applies it to molecular design.

Original: This paper is the ﬁrst work that applies distributed MCTS to a real-world and non-game problem.
Revised: This paper is the ﬁrst work that applies distributed MCTS to a real-world and non-game problem.

Original: Existing works on large-scale parallel MCTS show efﬁcient scalability in terms of the number of rollouts up to 100 workers.
Revised: Existing works on large-scale parallel MCTS show efﬁcient scalability in terms of the number of rollouts up to 100 workers.

Original: Still, they suffer from the degradation in the quality of the solutions.
Revised: Still, they suffer from the degradation in the quality of the solutions.

Original: MP-MCTS maintains the search quality at a larger scale.
Revised: MP-MCTS maintains the search quality at a larger scale.

Original: By running MP-MCTS on 256 CPU cores for only 10 minutes, we obtained candidate molecules with similar scores to non-parallel MCTS running for 42 hours.
Revised: By running MP-MCTS on 256 CPU cores for only 10 minutes, we obtained candidate molecules with similar scores to non-parallel MCTS running for 42 hours.

Original: Moreover, our results based on parallel MCTS (combined with a simple RNN model) signiﬁcantly outperform existing state-of-the-art work.
Revised: Moreover, our results based on parallel MCTS (combined with a simple RNN model) signiﬁcantly outperform existing state-of-the-art work.

Original: Our method is generic and is expected to speed up other applications of MCTS 1 .
Revised: Our method is generic and is expected to speed up other applications of MCTS 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A survey paper on MCTS, published in 2012, has cited 240 papers, including many game and non-game applications (Browne et al., 2012).
Revised: A survey paper on MCTS, published in 2012, has cited 240 papers, including many game and non-game applications (Browne et al., 2012).

Original: Since the invention of Upper Conﬁdence bound applied to Trees (UCT) (Kocsis & Szepesvári, 2006) (the most representative MCTS algorithm) in 2006,
Revised: Since the invention of Upper Conﬁdence bound applied to Trees (UCT) (Kocsis & Szepesvári, 2006) (the most representative MCTS algorithm) in 2006,

Original: MCTS has shown remarkable performance in various problems.
Revised: MCTS has shown remarkable performance in various problems.

Original: Recently, the successful combination with Deep Neural Networks (DNN) in computer Go by AlphaGo (Silver et al., 2016) has brought
Revised: Recently, the successful combination with Deep Neural Networks (DNN) in computer Go by AlphaGo (Silver et al., 2016) has brought

Original: MCTS into the spotlight.
Revised: MCTS into the spotlight.

Original: Combining MCTS and DNN is becoming one of the standard tools for solving decision making or combinatorial optimization problems.
Revised: Combining MCTS and DNN is becoming one of the standard tools for solving decision making or combinatorial optimization problems.

Original: Therefore, there is a signiﬁcant demand for parallel MCTS.
Revised: Therefore, there is a signiﬁcant demand for parallel MCTS.

Original: However, in contrast to the enormous computing resources invested in training DNN models in many recent studies, MCTS is rarely parallelized at large scale.
Revised: However, in contrast to the enormous computing resources invested in training DNN models in many recent studies, MCTS is rarely parallelized at large scale.

Original: Parallelizing MCTS/UCT is notoriously challenging.
Revised: Parallelizing MCTS/UCT is notoriously challenging.

Original: For example, in UCT, the algorithm follows four steps, selection-expansion-rollout-backpropagation.
Revised: For example, in UCT, the algorithm follows four steps, selection-expansion-rollout-backpropagation.

Original: Non-parallel vanilla UCT updates (backpropagates) the values in the tree nodes after each rollout.
Revised: Non-parallel vanilla UCT updates (backpropagates) the values in the tree nodes after each rollout.

Original: The behavior of the subsequent selection steps depends on the results of the previous rollouts-backpropagation.
Revised: The behavior of the subsequent selection steps depends on the results of the previous rollouts-backpropagation.

Original: Therefore, there is no apparent parallelism in the algorithm.
Revised: Therefore, there is no apparent parallelism in the algorithm.

Original: Using virtual-loss technique (explained in section 2.3), MCTS has been efﬁciently parallelized on shared-memory single machine environment, where the number of CPU cores are limited (Chaslot et al., 2008; Enzenberger & Müller, 2010; Segal, 2010).
Revised: Using virtual-loss technique (explained in section 2.3), MCTS has been efﬁciently parallelized on shared-memory single machine environment, where the number of CPU cores are limited (Chaslot et al., 2008; Enzenberger & Müller, 2010; Segal, 2010).

Original: However, there is limited research on large-scale parallel MCTS using distributed memory environments.
Revised: However, there is limited research on large-scale parallel MCTS using distributed memory environments.

Original: Only two approaches scale efﬁciently on distributed memory environment, but these were only validated in terms of the number of rollouts and the actual improvement is not validated (Yoshizoe et al., 2011; Graf et al., 2011).
Revised: Only two approaches scale efﬁciently on distributed memory environment, but these were only validated in terms of the number of rollouts and the actual improvement is not validated (Yoshizoe et al., 2011; Graf et al., 2011).

Original: Recently, the combination of (non-parallel) MCTS and DNN has been applied to molecular design problems, which aims to ﬁnd new chemical compounds with desired properties (Yang et al., 2017;
Revised: Recently, the combination of (non-parallel) MCTS and DNN has been applied to molecular design problems, which aims to ﬁnd new chemical compounds with desired properties (Yang et al., 2017;

Original: Sumita et al., 2018), utilizing the ability of MCTS to solve single-agent problems.
Revised: Sumita et al., 2018), utilizing the ability of MCTS to solve single-agent problems.

Original: In general, designing novel molecules can be formulated as a combinatorial optimization or planning problem to ﬁnd the optimal solutions in vast chemical space (of 10 23 to 10 60 , Polishchuk et al.
Revised: In general, designing novel molecules can be formulated as a combinatorial optimization or planning problem to ﬁnd the optimal solutions in vast chemical space (of 10 23 to 10 60 , Polishchuk et al.

Original: and can be tackled with the combinations of deep generative models and search (Kusner et al., 2017; GómezBombarelli et al., 2018; Jin et al., 2018;
Revised: and can be tackled with the combinations of deep generative models and search (Kusner et al., 2017; GómezBombarelli et al., 2018; Jin et al., 2018;

Original: Popova et al., 2018; 2019; Yang et al., 2017).
Revised: Popova et al., 2018; 2019; Yang et al., 2017).

Original: However, there are no previous studies about massively parallel MCTS for molecular design.
Revised: However, there are no previous studies about massively parallel MCTS for molecular design.

Original: In this paper, we propose a novel distributed parallel MCTS and apply it to the molecule design problem.
Revised: In this paper, we propose a novel distributed parallel MCTS and apply it to the molecule design problem.

Original: This is the ﬁrst work to explore viability of distributed parallel MCTS in molecular design.
Revised: This is the ﬁrst work to explore viability of distributed parallel MCTS in molecular design.

Original: Our experimental results show that a simple RNN model combined with massively parallel MCTS outperforms existing work using more complex models combined with Bayesian Optimization or
Revised: Our experimental results show that a simple RNN model combined with massively parallel MCTS outperforms existing work using more complex models combined with Bayesian Optimization or

Original: Reinforcement Learning (other than UCT).
Revised: Reinforcement Learning (other than UCT).

Original:  B ACKGROUND 2. ( NON - PARALLEL ) MCTS
Revised:  B ACKGROUND 2. ( NON - PARALLEL ) MCTS

Original: In 2006, Kocsis and Szepesvári proposed UCT based on a Multi-Armed Bandit algorithm UCB1 (Auer et al., 2002), which is the ﬁrst MCTS algorithm having a proof of convergence to the optimal solution.
Revised: In 2006, Kocsis and Szepesvári proposed UCT based on a Multi-Armed Bandit algorithm UCB1 (Auer et al., 2002), which is the ﬁrst MCTS algorithm having a proof of convergence to the optimal solution.

Original: It has shown good performance for many problems, including the game of Go (Gelly et al., 2006).
Revised: It has shown good performance for many problems, including the game of Go (Gelly et al., 2006).

Original: One round of UCT consists of four steps, as shown in Fig.
Revised: One round of UCT consists of four steps, as shown in Fig.

Original: It repeats the rounds for a given number of times or until a speciﬁed time has elapsed.
Revised: It repeats the rounds for a given number of times or until a speciﬁed time has elapsed.

Original: Selection:
Revised: Selection:

Original: The algorithm starts from the root node and selects the child with the highest UCB value (Fig. 2 left) until it reaches a leaf node.
Revised: The algorithm starts from the root node and selects the child with the highest UCB value (Fig. 2 left) until it reaches a leaf node.

Original: For each child i , v i is the number of visits, w i is the cumulative reward, and V is the number of visits at the parent node.
Revised: For each child i , v i is the number of visits, w i is the cumulative reward, and V is the number of visits at the parent node.

Original: Exploration constant C controls the behavior of UCT: the smaller, the more selective; the greater, the more explorative.
Revised: Exploration constant C controls the behavior of UCT: the smaller, the more selective; the greater, the more explorative.

Original: Expansion: If the number of visits exceeds a given threshold, expand the leaf node (add the children of the leaf to the tree) and select one of the new children for simulation.
Revised: Expansion: If the number of visits exceeds a given threshold, expand the leaf node (add the children of the leaf to the tree) and select one of the new children for simulation.

Original: Do nothing otherwise.
Revised: Do nothing otherwise.

Original: Simulation: UCT then evaluates the node by a simulation .
Revised: Simulation: UCT then evaluates the node by a simulation .

Original: This step is often called playout or rollout.
Revised: This step is often called playout or rollout.

Original: A simple example of rollout is to go down the tree by selecting a random child at each node until it reaches a terminal node and returns the value at the terminal node as the reward r (win or loss for games).
Revised: A simple example of rollout is to go down the tree by selecting a random child at each node until it reaches a terminal node and returns the value at the terminal node as the reward r (win or loss for games).

Original: Replacing rollout with a DNN based evaluation is becoming more popular following the success of AlphaGo.
Revised: Replacing rollout with a DNN based evaluation is becoming more popular following the success of AlphaGo.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: SMILES
Revised: SMILES

Original: Complete SMILES
Revised: Complete SMILES

Original: Score(reward)
Revised: Score(reward)

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: n&
Revised: n&

Original: O
Revised: O

Original: C=
Revised: C=

Original: c
Revised: c

Original: C
Revised: C

Original: Nsn&
Revised: Nsn&

Original: O
Revised: O

Original: C=
Revised: C=

Original: c
Revised: c

Original: C
Revised: C

Original: Nsn
Revised: Nsn

Original: [Figure]
Revised: [Figure]

Original: Simulation
Revised: Simulation

Original: [Figure]
Revised: [Figure]

Original: RNNrollout
Revised: RNNrollout

Original: Chemistrysimulator
Revised: Chemistrysimulator

Original: O N N Cl
Revised: O N N Cl

Original: N
Revised: N

Original: Cl Cl
Revised: Cl Cl

Original: O F F non-parallel) MCTS, with simulation for molecular design.
Revised: O F F non-parallel) MCTS, with simulation for molecular design.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Green worker goes first, increments t i on the path and reaches a leaf.
Revised: Green worker goes first, increments t i on the path and reaches a leaf.

Original: Red goes second, increments t i and avoids the
Revised: Red goes second, increments t i and avoids the

Original: : UCT ﬁnally traverses the path all the way back to the root, and updates the values of the nodes in the path ( w i =
Revised: : UCT ﬁnally traverses the path all the way back to the root, and updates the values of the nodes in the path ( w i =

Original: w i + r , v i
Revised: w i + r , v i

Original: = v i + 1 )
Revised: = v i + 1 )

Original: 2. S OLVING M OLECULAR D ESIGN USING MCTS
Revised: 2. S OLVING M OLECULAR D ESIGN USING MCTS

Original: The Simpliﬁed Molecular-Input Line-Entry System (SMILES)
Revised: The Simpliﬁed Molecular-Input Line-Entry System (SMILES)

Original: (Weininger, 1988) is a standard notation in chemistry for describing molecules using ASCII strings, deﬁned by a grammar.
Revised: (Weininger, 1988) is a standard notation in chemistry for describing molecules using ASCII strings, deﬁned by a grammar.

Original: SMILES uses ASCII symbols to denote atoms, bonds, or structural information such as rings.
Revised: SMILES uses ASCII symbols to denote atoms, bonds, or structural information such as rings.

Original: For example,
Revised: For example,

Original: SMILES for Carbon dioxide is "O=C
Revised: SMILES for Carbon dioxide is "O=C

Original: =O" where "=" means a double bond, and for Benzene, it is "C1=CC=CC=C1" or "c1ccccc1" which forms a ring by connecting the two "C1"s.
Revised: =O" where "=" means a double bond, and for Benzene, it is "C1=CC=CC=C1" or "c1ccccc1" which forms a ring by connecting the two "C1"s.

Original: The search space for molecular design can be deﬁned as a tree based on the SMILES, where the root node is the starting symbol (denoted as "&", usually not shown) and each path from the root to a depth d node represents a left-to-right enumeration of the ﬁrst d symbols in the (partial) SMILES string (see Fig.
Revised: The search space for molecular design can be deﬁned as a tree based on the SMILES, where the root node is the starting symbol (denoted as "&", usually not shown) and each path from the root to a depth d node represents a left-to-right enumeration of the ﬁrst d symbols in the (partial) SMILES string (see Fig.

Original: The subscripts shows the depth.).
Revised: The subscripts shows the depth.).

Original: Yang et al.
Revised: Yang et al.

Original: (2017) were the ﬁrst to apply non-parallel MCTS to this problem using the AlphaGo-like approach, which combines MCTS with DNN and a computational chemistry simulator.
Revised: (2017) were the ﬁrst to apply non-parallel MCTS to this problem using the AlphaGo-like approach, which combines MCTS with DNN and a computational chemistry simulator.

Original: They trained an RNN model with a chemical compound database, and the RNN predicts the next symbol of a partial SMILES string which is given as the input.
Revised: They trained an RNN model with a chemical compound database, and the RNN predicts the next symbol of a partial SMILES string which is given as the input.

Original: The model is used for expansion and rollout.
Revised: The model is used for expansion and rollout.

Original: Expansion for Molecular Design:
Revised: Expansion for Molecular Design:

Original: The node ( = 2 ) in Fig.
Revised: The node ( = 2 ) in Fig.

Original: 1 denotes SMILES strings starting with "O=".
Revised: 1 denotes SMILES strings starting with "O=".

Original: The RNN receives "O=" as input and outputs the probability of the next SMILES symbols.
Revised: The RNN receives "O=" as input and outputs the probability of the next SMILES symbols.

Original: Instead of simply adding all symbols as child nodes, the low probability branches are pruned based on the output of the model.
Revised: Instead of simply adding all symbols as child nodes, the low probability branches are pruned based on the output of the model.

Original: Simulation for Molecular Design: Fig.
Revised: Simulation for Molecular Design: Fig.

Original: 1 illustrates the simulation step for molecular design.
Revised: 1 illustrates the simulation step for molecular design.

Original: Node ( C 3 ) denotes SMILES strings starting with "O=C".
Revised: Node ( C 3 ) denotes SMILES strings starting with "O=C".

Original: Firstly, a rollout generates a complete SMILES string, a new candidate molecule, by repeatedly sampling the next symbol using the model, until the model outputs the terminal symbol.
Revised: Firstly, a rollout generates a complete SMILES string, a new candidate molecule, by repeatedly sampling the next symbol using the model, until the model outputs the terminal symbol.

Original: Then, a computational chemistry simulator receives the SMILES string and calculates the target chemical property for the molecule, to determine the reward.
Revised: Then, a computational chemistry simulator receives the SMILES string and calculates the target chemical property for the molecule, to determine the reward.

Original: 2. P ARALLEL MCTS
Revised: 2. P ARALLEL MCTS

Original: Non-parallel MCTS ﬁnds the most promising leaf one at a time in the selection step.
Revised: Non-parallel MCTS ﬁnds the most promising leaf one at a time in the selection step.

Original: In parallel
Revised: In parallel

Original: MCTS, multiple workers must ﬁnd the promising leaves (to launch simulations from) in parallel, hence, should ﬁnd leaves speculatively without knowing the latest results.
Revised: MCTS, multiple workers must ﬁnd the promising leaves (to launch simulations from) in parallel, hence, should ﬁnd leaves speculatively without knowing the latest results.

Original: Fig.
Revised: Fig.

Original: 2 shows the example of a three workers case (a worker means a thread or a process, which runs on either a CPU or GPU).
Revised: 2 shows the example of a three workers case (a worker means a thread or a process, which runs on either a CPU or GPU).

Original: If the workers (shown in green, red, blue) follow the UCB1 formula, all end up at the same leaf node (Fig. 2-(a)), and the parallelization fails.
Revised: If the workers (shown in green, red, blue) follow the UCB1 formula, all end up at the same leaf node (Fig. 2-(a)), and the parallelization fails.

Original: The Virtual loss (Chaslot et al., 2008) let the workers ﬁnd the promising but different leaves.
Revised: The Virtual loss (Chaslot et al., 2008) let the workers ﬁnd the promising but different leaves.

Original: The original UCB1 is modiﬁed to UCB vl shown in Fig. 2-(b), where t i is the number of workers currently searching in the subtree of the child node i , and T is the sum of t i of the children.
Revised: The original UCB1 is modiﬁed to UCB vl shown in Fig. 2-(b), where t i is the number of workers currently searching in the subtree of the child node i , and T is the sum of t i of the children.

Original: It is called virtual loss because it assumes that the current ongoing simulations will obtain 0 reward.
Revised: It is called virtual loss because it assumes that the current ongoing simulations will obtain 0 reward.

Original: With this modiﬁcation, UCB vl value is penalized based on the number of workers, which makes the subsequent workers avoid the subtrees already being explored (see Fig. 2-(b)).
Revised: With this modiﬁcation, UCB vl value is penalized based on the number of workers, which makes the subsequent workers avoid the subtrees already being explored (see Fig. 2-(b)).

Original: [Figure]
Revised: [Figure]

Original: Parallel UCT with virtual loss was proved to improve the strength of shared-memory (using up tocores)
Revised: Parallel UCT with virtual loss was proved to improve the strength of shared-memory (using up tocores)

Original: Go programs (Chaslot et al., 2008; Enzenberger & Müller, 2010).
Revised: Go programs (Chaslot et al., 2008; Enzenberger & Müller, 2010).

Original: Segal’s experiment on an emulated multithreaded machine that assumes no communication overhead shows speedup on up to 512 threads.
Revised: Segal’s experiment on an emulated multithreaded machine that assumes no communication overhead shows speedup on up to 512 threads.

Original: However, his experiments on real distributed machines did not provide signiﬁcant speedup beyond 32 cores (Segal, 2010).
Revised: However, his experiments on real distributed machines did not provide signiﬁcant speedup beyond 32 cores (Segal, 2010).

Original: 2. H ASH - DRIVEN P ARALLEL S EARCH AND U NIFORM L OAD B ALANCING
Revised: 2. H ASH - DRIVEN P ARALLEL S EARCH AND U NIFORM L OAD B ALANCING

Original: When many workers perform a search in parallel, it is essential to distribute the search space as evenly as possible.
Revised: When many workers perform a search in parallel, it is essential to distribute the search space as evenly as possible.

Original: If there is an imbalance in the workload, overall performance will be less efﬁcient because only some workers will continue the computation while the rest remain idle.
Revised: If there is an imbalance in the workload, overall performance will be less efﬁcient because only some workers will continue the computation while the rest remain idle.

Original: If all the search space is explicitly given in advance, this is a trivial matter.
Revised: If all the search space is explicitly given in advance, this is a trivial matter.

Original: However, in case of problems such as games and combinatorial optimization, where the search space is generated while the search is performed, it is difﬁcult to distribute the search space evenly.
Revised: However, in case of problems such as games and combinatorial optimization, where the search space is generated while the search is performed, it is difﬁcult to distribute the search space evenly.

Original: Hash-driven (HD) parallel search is one of the methods for resolving this difﬁculty; Evett et al.
Revised: Hash-driven (HD) parallel search is one of the methods for resolving this difﬁculty; Evett et al.

Original: (1995) (PRA*) and Romein et al.
Revised: (1995) (PRA*) and Romein et al.

Original: (1999) (TDS: Transposition-table Driven Scheduling) developed this method independently and applied to parallel Iterative Deepening A* search.
Revised: (1999) (TDS: Transposition-table Driven Scheduling) developed this method independently and applied to parallel Iterative Deepening A* search.

Original: Kishimoto et al.
Revised: Kishimoto et al.

Original: (2013) later applied it to parallel A* search.
Revised: (2013) later applied it to parallel A* search.

Original: HD parallel search requires a hash function that deﬁnes the hash key of each node.
Revised: HD parallel search requires a hash function that deﬁnes the hash key of each node.

Original: Each time the algorithm creates a node during the search, it assigns the node to a speciﬁc worker, the home worker (or the home processor) , based on the hash function, and achieves a near-equal load balancing if the hash function is sufﬁciently random (such as Zobrist
Revised: Each time the algorithm creates a node during the search, it assigns the node to a speciﬁc worker, the home worker (or the home processor) , based on the hash function, and achieves a near-equal load balancing if the hash function is sufﬁciently random (such as Zobrist

Original: Hashing (Zobrist, 1970)).
Revised: Hashing (Zobrist, 1970)).

Original: Figure 3 illustrates this method.
Revised: Figure 3 illustrates this method.

Original: The hash function randomly divides the tree into four partitions (shown in four colors), and each worker holds the nodes in the assigned partition in its hash table.
Revised: The hash function randomly divides the tree into four partitions (shown in four colors), and each worker holds the nodes in the assigned partition in its hash table.

Original: A distinct drawback of this method is that it requires communication between workers almost every time the algorithm traverses a branch (unless the two nodes are on the same worker).
Revised: A distinct drawback of this method is that it requires communication between workers almost every time the algorithm traverses a branch (unless the two nodes are on the same worker).

Original: The key to efﬁcient parallel speedup is the trade-off between uniform load balancing and frequent communication.
Revised: The key to efﬁcient parallel speedup is the trade-off between uniform load balancing and frequent communication.

Original: The experimental results for IDA* and A* (Evett et al., 1995; Romein et al., 1999; Kishimoto et al., 2013) prove its efﬁciency for these search algorithms.
Revised: The experimental results for IDA* and A* (Evett et al., 1995; Romein et al., 1999; Kishimoto et al., 2013) prove its efﬁciency for these search algorithms.

Original: However, a straightforward application of HD parallel search to UCT is not efﬁcient enough as described below.
Revised: However, a straightforward application of HD parallel search to UCT is not efﬁcient enough as described below.

Original:  D ISTRIBUTED P ARALLEL MCTS AND M OLECULAR D ESIGN 3. TDS-UCT: H ASH - DRIVEN P ARALLEL MCTS
Revised:  D ISTRIBUTED P ARALLEL MCTS AND M OLECULAR D ESIGN 3. TDS-UCT: H ASH - DRIVEN P ARALLEL MCTS

Original: Here we describe the application of Hash-driven parallel search to
Revised: Here we describe the application of Hash-driven parallel search to

Original: UCT
Revised: UCT

Original: , TDS-UCT.
Revised: , TDS-UCT.

Original: Figure 4-(a) illustrates the behavior of the four steps in TDS-UCT.
Revised: Figure 4-(a) illustrates the behavior of the four steps in TDS-UCT.

Original: Selection: The worker which holds the root node selects the best child ( O 1 in Fig. 4) based on UCB vl formula (Fig. 2-(b)).
Revised: Selection: The worker which holds the root node selects the best child ( O 1 in Fig. 4) based on UCB vl formula (Fig. 2-(b)).

Original: It then sends a selection message , which holds information of O 1 , to the home worker of O 1 (the green worker in Fig.4).
Revised: It then sends a selection message , which holds information of O 1 , to the home worker of O 1 (the green worker in Fig.4).

Original: If a worker receives a selection message , it selects the best child of the node and pass the message to another worker until the message reaches a leaf node.
Revised: If a worker receives a selection message , it selects the best child of the node and pass the message to another worker until the message reaches a leaf node.

Original: The worker-count t i of each node is incremented during the step.
Revised: The worker-count t i of each node is incremented during the step.

Original: Expansion: If a worker receives a selection message and the node is a leaf, the expansion step is done in the same way as in non-parallel MCTS.
Revised: Expansion: If a worker receives a selection message and the node is a leaf, the expansion step is done in the same way as in non-parallel MCTS.

Original: Simulation: Simulation is done by the home worker of the leaf (the green worker in Fig. 4) in the same way as in non-parallel MCTS.
Revised: Simulation: Simulation is done by the home worker of the leaf (the green worker in Fig. 4) in the same way as in non-parallel MCTS.

Original: Backpropagation: After a simulation (at C 3 ), a backpropagation message , which holds the reward r , is sent to the parent ( = 2 ).
Revised: Backpropagation: After a simulation (at C 3 ), a backpropagation message , which holds the reward r , is sent to the parent ( = 2 ).

Original: The workers pass the backpropagation message to the parent until it reaches the root node.
Revised: The workers pass the backpropagation message to the parent until it reaches the root node.

Original: At each node, the values are updated by the corresponding worker ( w i =
Revised: At each node, the values are updated by the corresponding worker ( w i =

Original: w i + r , v i
Revised: w i + r , v i

Original: = v i + 1 , t i
Revised: = v i + 1 , t i

Original: =
Revised: =

Original: t i − 1 ).
Revised: t i − 1 ).

Original: To reduce the the number of idle workers, the sum of the number of selection messages , backpropagation messages and ongoing simulations must be more than the number of workers.
Revised: To reduce the the number of idle workers, the sum of the number of selection messages , backpropagation messages and ongoing simulations must be more than the number of workers.

Original: Yoshizoe et al.
Revised: Yoshizoe et al.

Original: (2011) and Graf et al.
Revised: (2011) and Graf et al.

Original: (2011) independently proposed to control the number to N × # workers where N is the overload factor .
Revised: (2011) independently proposed to control the number to N × # workers where N is the overload factor .

Original: We used N = 3 following the experimental results in the two papers.
Revised: We used N = 3 following the experimental results in the two papers.

Original: 3. TDS- DF -UCT: D EPTH -F IRST R EFORMULATION OF TDS-UCT
Revised: 3. TDS- DF -UCT: D EPTH -F IRST R EFORMULATION OF TDS-UCT

Original: Scalability of TDS-UCT is limited because of the communication contention around the root node.
Revised: Scalability of TDS-UCT is limited because of the communication contention around the root node.

Original: In TDS-UCT, all backpropagation messages are sent up until to the root node.
Revised: In TDS-UCT, all backpropagation messages are sent up until to the root node.

Original: As the number of messages increase, the workers that hold shallow nodes (especially the root node) spend more time for communication.
Revised: As the number of messages increase, the workers that hold shallow nodes (especially the root node) spend more time for communication.

Original: Because of this problem, the scalability of TDS-UCT quickly diminishes beyond 100 workers (Yoshizoe et al., 2011).
Revised: Because of this problem, the scalability of TDS-UCT quickly diminishes beyond 100 workers (Yoshizoe et al., 2011).

Original: For solving the communication contention, TDS-df-UCT was proposed based on the observation that the promising part of the tree does not change so often after each simulation (Yoshizoe et al., 2011).
Revised: For solving the communication contention, TDS-df-UCT was proposed based on the observation that the promising part of the tree does not change so often after each simulation (Yoshizoe et al., 2011).

Original: The next selection step will likely reach a node that is close to the previously selected leaf node, as shown in Fig. 4-(a).
Revised: The next selection step will likely reach a node that is close to the previously selected leaf node, as shown in Fig. 4-(a).

Original: The leaf is C 3 for the 1st selection step and N 3 for the 2nd selection step.
Revised: The leaf is C 3 for the 1st selection step and N 3 for the 2nd selection step.

Original: Each job message in TDS-df-UCT carries a history table which contains the history of the siblings of the nodes in the current path.
Revised: Each job message in TDS-df-UCT carries a history table which contains the history of the siblings of the nodes in the current path.

Original: After each simulation, TDS-df-UCT updates the values in the node and the history table, and re-calculate the UCB vl of the siblings.
Revised: After each simulation, TDS-df-UCT updates the values in the node and the history table, and re-calculate the UCB vl of the siblings.

Original: Unlike TDS-UCT, TDS-df-UCT will backpropagate only if the UCB vl value of the nodes in the current path is exceeded by one of the other siblings in the history table.
Revised: Unlike TDS-UCT, TDS-df-UCT will backpropagate only if the UCB vl value of the nodes in the current path is exceeded by one of the other siblings in the history table.

Original: Fig.
Revised: Fig.

Original: 4-(b) shows a trajectory of a message in TDS-df-UCT.
Revised: 4-(b) shows a trajectory of a message in TDS-df-UCT.

Original: After the ﬁrst selection-expansionsimulation, it does not send the backpropagation message to the parent of = 2 .
Revised: After the ﬁrst selection-expansionsimulation, it does not send the backpropagation message to the parent of = 2 .

Original: Instead, it directly sends a selection message to N 3 , as = 2 is still the most promising node.
Revised: Instead, it directly sends a selection message to N 3 , as = 2 is still the most promising node.

Original: This technique dramatically reduces the number of messages sent to the root node by staying at the more promising part, thus solving the communication contention problem.
Revised: This technique dramatically reduces the number of messages sent to the root node by staying at the more promising part, thus solving the communication contention problem.

Original: 3. MP-MCTS: ADDRESSING SHORTCOMINGS OF TDS- DF -UCT
Revised: 3. MP-MCTS: ADDRESSING SHORTCOMINGS OF TDS- DF -UCT

Original: TDS-df-UCT carries the history only in the messages, hence, requires more time to propagate new results to other jobs.
Revised: TDS-df-UCT carries the history only in the messages, hence, requires more time to propagate new results to other jobs.

Original: As a result, TDS-df-UCT skips backpropagations too often because it overlooks that other workers may have found more promising part of the tree, therefore making the tree wider and shallower.
Revised: As a result, TDS-df-UCT skips backpropagations too often because it overlooks that other workers may have found more promising part of the tree, therefore making the tree wider and shallower.

Original: In our MP-MCTS, along with its own statistical information ( w , v , t ), each node maintains the history table as shown in Fig.
Revised: In our MP-MCTS, along with its own statistical information ( w , v , t ), each node maintains the history table as shown in Fig.

Original: 4 (c).
Revised: 4 (c).

Original: It accelerates the propagation of the latest information at the cost of extra (minor) memory usage (see Appendix A.2 for the memory usage analysis).
Revised: It accelerates the propagation of the latest information at the cost of extra (minor) memory usage (see Appendix A.2 for the memory usage analysis).

Original: This modiﬁcation to TDS-df-UCT is crucial for improving the behavior of the parallel
Revised: This modiﬁcation to TDS-df-UCT is crucial for improving the behavior of the parallel

Original: MCTS to generate a deeper tree.
Revised: MCTS to generate a deeper tree.

Original: We illustrate in the experimental results that the quality of the
Revised: We illustrate in the experimental results that the quality of the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: (c) MP
Revised: (c) MP

Original: [Equation]
Revised: [Equation]

Original: (O
Revised: (O

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: t (= 2 ) w
Revised: t (= 2 ) w

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: t (c 2 ) w
Revised: t (c 2 ) w

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: t (F 2 ) w, v, t (C 3 ) w, v, t (N 3 ) w, v, t &
Revised: t (F 2 ) w, v, t (C 3 ) w, v, t (N 3 ) w, v, t &

Original: O
Revised: O

Original: C=
Revised: C=

Original: c
Revised: c

Original: C
Revised: C

Original: Nsn 2 F&
Revised: Nsn 2 F&

Original: O
Revised: O

Original: C=
Revised: C=

Original: c
Revised: c

Original: C
Revised: C

Original: Nsn 2 F(O 1 ) w, v, t (C 1 ) w, v, t (= 2 ) w, v, t (c 2 ) w, v, t (F 2 ) w, v, t (C 3 ) w, v, t (N 3 ) w, v, t solution and the average depth of the tree generated by N -parallel MP-MCTS are both similar to the ideal case obtained by running the non-parallel MCTS for N -times more wall-clock time.
Revised: Nsn 2 F(O 1 ) w, v, t (C 1 ) w, v, t (= 2 ) w, v, t (c 2 ) w, v, t (F 2 ) w, v, t (C 3 ) w, v, t (N 3 ) w, v, t solution and the average depth of the tree generated by N -parallel MP-MCTS are both similar to the ideal case obtained by running the non-parallel MCTS for N -times more wall-clock time.

Original:  E XPERIMENT M ETHODOLOGY
Revised:  E XPERIMENT M ETHODOLOGY

Original: To evaluate the quality of the solutions and to compare against state-of-the-art methods, we use the octanol-water partition coefﬁcient (logP) penalized by the synthetic accessibility (SA) and
Revised: To evaluate the quality of the solutions and to compare against state-of-the-art methods, we use the octanol-water partition coefﬁcient (logP) penalized by the synthetic accessibility (SA) and

Original: large Ring
Revised: large Ring

Original: Penalty score, a popular benchmarking physicochemical property (Kusner et al., 2017; Jin et al., 2018;
Revised: Penalty score, a popular benchmarking physicochemical property (Kusner et al., 2017; Jin et al., 2018;

Original: You et al., 2018a; Popova et al., 2019; Maziarka et al., 2020) used in molecular design.
Revised: You et al., 2018a; Popova et al., 2019; Maziarka et al., 2020) used in molecular design.

Original: Model and Dataset: We use the GRU-based model and pre-trained weights publicly available on the repository of ChemTS (Yang et al., 2017), which mainly consists of two stacked GRU units.
Revised: Model and Dataset: We use the GRU-based model and pre-trained weights publicly available on the repository of ChemTS (Yang et al., 2017), which mainly consists of two stacked GRU units.

Original: Input data represents SMILES symbols using 64-dim one-hot vectors.
Revised: Input data represents SMILES symbols using 64-dim one-hot vectors.

Original: The ﬁrst GRU layer has 64-dim input/256-dim output.
Revised: The ﬁrst GRU layer has 64-dim input/256-dim output.

Original: The second GRU layer has 256-dim input/256-dim output, connected to the last dense layer, which outputs 64 values with softmax.
Revised: The second GRU layer has 256-dim input/256-dim output, connected to the last dense layer, which outputs 64 values with softmax.

Original: The model was pre-trained using a molecule dataset that contains 250K drug molecules extracted from the ZINC database, following (Kusner et al., 2017; Yang et al., 2017; Jin et al., 2018).
Revised: The model was pre-trained using a molecule dataset that contains 250K drug molecules extracted from the ZINC database, following (Kusner et al., 2017; Yang et al., 2017; Jin et al., 2018).

Original: MCTS implementation: The GRU model explained above takes a partial SMILES string as the input and outputs the probability of the next symbol.
Revised: MCTS implementation: The GRU model explained above takes a partial SMILES string as the input and outputs the probability of the next symbol.

Original: We use this model for both Expansion and Simulation.
Revised: We use this model for both Expansion and Simulation.

Original: In the expansion step, we add branches (e.g., SMILES symbols) with high probability until the cumulative probability reaches 0.95.
Revised: In the expansion step, we add branches (e.g., SMILES symbols) with high probability until the cumulative probability reaches 0.95.

Original: For a rollout in the simulation step, we repeatedly sample over the model to generate a complete SMILES string.
Revised: For a rollout in the simulation step, we repeatedly sample over the model to generate a complete SMILES string.

Original: Then the string is passed to a computational chemistry tool, RDKit (Landrum et al., 2006) for calculating the penalized logP score, which is commonly used in existing work.
Revised: Then the string is passed to a computational chemistry tool, RDKit (Landrum et al., 2006) for calculating the penalized logP score, which is commonly used in existing work.

Original: We use the reward deﬁnition described in Yang et al.
Revised: We use the reward deﬁnition described in Yang et al.

Original: (Yang et al., 2017) which is normalized to [ − 1 , 1] and consider the same value for exploration constant, C = 1 .
Revised: (Yang et al., 2017) which is normalized to [ − 1 , 1] and consider the same value for exploration constant, C = 1 .

Original: Other experimental settings: Algorithms are implemented using Keras with TensorFlow and MPI library for Python (mpi4py).
Revised: Other experimental settings: Algorithms are implemented using Keras with TensorFlow and MPI library for Python (mpi4py).

Original: All experiments, unless otherwise speciﬁed, were run for 10 minutes on up to 1024 cores of a CPU cluster (each node equipped with two Intel Xeon Gold 6148 CPU (2.4GHz, 20 cores) and 384GB of memory), and one MPI process (called worker in this paper) is assigned to one core.
Revised: All experiments, unless otherwise speciﬁed, were run for 10 minutes on up to 1024 cores of a CPU cluster (each node equipped with two Intel Xeon Gold 6148 CPU (2.4GHz, 20 cores) and 384GB of memory), and one MPI process (called worker in this paper) is assigned to one core.

Original:  E XPERIMENT R ESULTS
Revised:  E XPERIMENT R ESULTS

Original: To evaluate the distributed MCTS approaches, we study the quality of the solutions obtained, analyze the performance of different parallel MCTS algorithms, and compare the performance of MP-MCTS with state-of-the-art work in molecular design.
Revised: To evaluate the distributed MCTS approaches, we study the quality of the solutions obtained, analyze the performance of different parallel MCTS algorithms, and compare the performance of MP-MCTS with state-of-the-art work in molecular design.

Original: Maximizing penalized logP score.
Revised: Maximizing penalized logP score.

Original: Table 1 presents the penalized logP score of distributed MCTS approaches for varying number of CPU cores.
Revised: Table 1 presents the penalized logP score of distributed MCTS approaches for varying number of CPU cores.

Original: With increasing number of cores available, more number of simulations can be performed in parallel, which improves the quality of the score.
Revised: With increasing number of cores available, more number of simulations can be performed in parallel, which improves the quality of the score.

Original: We performed 10 runs for each settings, and show the average and standard deviation of the best scores.
Revised: We performed 10 runs for each settings, and show the average and standard deviation of the best scores.

Original: TDS-UCT suffers from communication contention, with increasing number of cores the load imbalance becomes more signiﬁcant, hence yields lower score.
Revised: TDS-UCT suffers from communication contention, with increasing number of cores the load imbalance becomes more signiﬁcant, hence yields lower score.

Original: TDS-df-UCT achieves more uniform load balancing but only slightly improves the score (discussed later in this section).
Revised: TDS-df-UCT achieves more uniform load balancing but only slightly improves the score (discussed later in this section).

Original: Our MP-MCTS, which mitigates the issues of other approaches, shows strict improvement in score with increase in number of cores leveraged.
Revised: Our MP-MCTS, which mitigates the issues of other approaches, shows strict improvement in score with increase in number of cores leveraged.

Original: Quality of parallel solution over non-parallel solution.
Revised: Quality of parallel solution over non-parallel solution.

Original: As mentioned earlier, any parallel MCTS must speculatively start to search before knowing the latest search results, and it may return different outcomes from those of the non-parallel version.
Revised: As mentioned earlier, any parallel MCTS must speculatively start to search before knowing the latest search results, and it may return different outcomes from those of the non-parallel version.

Original: In addition, the exploration and exploitation trade-off of distributed MCTS is controlled via the virtual-loss based UCB vl instead of theoretically guaranteed
Revised: In addition, the exploration and exploitation trade-off of distributed MCTS is controlled via the virtual-loss based UCB vl instead of theoretically guaranteed

Original: UCB1 (Browne et al., 2012).
Revised: UCB1 (Browne et al., 2012).

Original: Hence, it is signiﬁcant to compare the quality of distributed MCTS solution with non-parallel MCTS (cf.
Revised: Hence, it is signiﬁcant to compare the quality of distributed MCTS solution with non-parallel MCTS (cf.

Original: strength speedup (Appendix A.1)).
Revised: strength speedup (Appendix A.1)).

Original: The bottom row of Table 1 presents the penalized logP score for non-parallel MCTS.
Revised: The bottom row of Table 1 presents the penalized logP score for non-parallel MCTS.

Original: Note that non-parallel MCTS was run for equivalent core-hours (for example, 256 cores for non-parallel MCTS indicates it was run for 256 × 10 minutes on a single core; while distributed MCTS is run on 256 cores
Revised: Note that non-parallel MCTS was run for equivalent core-hours (for example, 256 cores for non-parallel MCTS indicates it was run for 256 × 10 minutes on a single core; while distributed MCTS is run on 256 cores

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: reduction at root.
Revised: reduction at root.

Original: Fig.
Revised: Fig.

Original: 5 (a) presents back-propagation messages received by each worker.
Revised: 5 (a) presents back-propagation messages received by each worker.

Original: MP-MCTS and TDS-df-UCT lead to better load balance across all workers, while for
Revised: MP-MCTS and TDS-df-UCT lead to better load balance across all workers, while for

Original: TDS-UCT, root’s and few other hot nodes’ home worker get overwhelmed by the messages, and suffers from extremely high load-imbalance.
Revised: TDS-UCT, root’s and few other hot nodes’ home worker get overwhelmed by the messages, and suffers from extremely high load-imbalance.

Original: Note that TDS-df-UCT generates a shallower tree, resulting in fewer BP messages.
Revised: Note that TDS-df-UCT generates a shallower tree, resulting in fewer BP messages.

Original: Depth of the search tree.
Revised: Depth of the search tree.

Original: Fig.
Revised: Fig.

Original: 5 (b) compares the depth of the nodes in the search tree generated by parallel and non-parallel MCTS.
Revised: 5 (b) compares the depth of the nodes in the search tree generated by parallel and non-parallel MCTS.

Original: TDS-UCT creates shallower tree than the others because of the load imbalance.
Revised: TDS-UCT creates shallower tree than the others because of the load imbalance.

Original: TDS-df-UCT generates a deeper tree than TDS-UCT but still much shallower than ideal case.
Revised: TDS-df-UCT generates a deeper tree than TDS-UCT but still much shallower than ideal case.

Original: It is presumed that TDS-df-UCT does too few backpropagations (see 3.2).
Revised: It is presumed that TDS-df-UCT does too few backpropagations (see 3.2).

Original: MP-MCTS generates a signiﬁcantly deeper tree by using 256 cores for 10 minutes, which is close to the ideal case of non-parallel MCTS using 2,560 minutes (approx. 42h).
Revised: MP-MCTS generates a signiﬁcantly deeper tree by using 256 cores for 10 minutes, which is close to the ideal case of non-parallel MCTS using 2,560 minutes (approx. 42h).

Original: Comparison against related work in molecular design.
Revised: Comparison against related work in molecular design.

Original: Table 2 presents the top 3 penalized logP scores obtained by the existing state-of-the-art work (description in section 6).
Revised: Table 2 presents the top 3 penalized logP scores obtained by the existing state-of-the-art work (description in section 6).

Original: The scores of
Revised: The scores of

Original: MP-MCTS are the best among 10 runs (10 minutes each), which outperforms the existing work signiﬁcantly in maximizing the penalized logP score.
Revised: MP-MCTS are the best among 10 runs (10 minutes each), which outperforms the existing work signiﬁcantly in maximizing the penalized logP score.

Original: It is also notable that MP-MCTS signiﬁcantly improved the score of the GRU-based model (Yang et al., 2017).
Revised: It is also notable that MP-MCTS signiﬁcantly improved the score of the GRU-based model (Yang et al., 2017).

Original: The bottom two lines compare the results obtained from 10 minutes random sampling from the GRU-based model with score obtained by MP-MCTS which uses the same model (as mentioned in Section 4).
Revised: The bottom two lines compare the results obtained from 10 minutes random sampling from the GRU-based model with score obtained by MP-MCTS which uses the same model (as mentioned in Section 4).

Original: This result suggests the possibility of improving existing work by combining their models with parallel MCTS.
Revised: This result suggests the possibility of improving existing work by combining their models with parallel MCTS.

Original: [Table]
Revised: [Table]

Original:  R ELATED WORK 6. M OLECULAR D ESIGN
Revised:  R ELATED WORK 6. M OLECULAR D ESIGN

Original: In molecular design
Revised: In molecular design

Original: [Table]
Revised: [Table]

Original: it is a common approach to use deep generative models (to generate candidate molecules)
Revised: it is a common approach to use deep generative models (to generate candidate molecules)

Original: [Table]
Revised: [Table]

Original: followed by optimization algorithms to focus on the promising candidates having desired molecular property (mainly Bayesian Optimization (BO) or Reinforcement learning (RL)).
Revised: followed by optimization algorithms to focus on the promising candidates having desired molecular property (mainly Bayesian Optimization (BO) or Reinforcement learning (RL)).

Original: GomezBombarelli et al.
Revised: GomezBombarelli et al.

Original: (Gómez-Bombarelli et al., 2018) were the ﬁrst to employ variational autoencoders (VAE).
Revised: (Gómez-Bombarelli et al., 2018) were the ﬁrst to employ variational autoencoders (VAE).

Original: Kusner et al.
Revised: Kusner et al.

Original: (Kusner et al., 2017) enhanced it to grammar variational autoencoder (GVAE) by combining context free grammars.
Revised: (Kusner et al., 2017) enhanced it to grammar variational autoencoder (GVAE) by combining context free grammars.

Original: Both of the above used BO for optimization.
Revised: Both of the above used BO for optimization.

Original: Segler et al.
Revised: Segler et al.

Original: (Segler et al., 2018a) focused on generating molecules using LSTM (Hochreiter & Schmidhuber, 1997).
Revised: (Segler et al., 2018a) focused on generating molecules using LSTM (Hochreiter & Schmidhuber, 1997).

Original: Olivecrona et al.
Revised: Olivecrona et al.

Original: (Olivecrona et al., 2017) used simple RNN and Popova et al.
Revised: (Olivecrona et al., 2017) used simple RNN and Popova et al.

Original: (Popova et al., 2018) used GRU for generation, both combined with RL.
Revised: (Popova et al., 2018) used GRU for generation, both combined with RL.

Original: These work use SMILES representations.
Revised: These work use SMILES representations.

Original: Graphbased molecule generation, such as JT-VAE (Jin et al., 2018) and GCPN (You et al., 2018a) generate molecules by directly operating on molecular graphs, optimized with BO and RL respectively.
Revised: Graphbased molecule generation, such as JT-VAE (Jin et al., 2018) and GCPN (You et al., 2018a) generate molecules by directly operating on molecular graphs, optimized with BO and RL respectively.

Original: Popova et al.
Revised: Popova et al.

Original: (Popova et al., 2019) proposed MolecularRNN, an improved RNN model by extending the
Revised: (Popova et al., 2019) proposed MolecularRNN, an improved RNN model by extending the

Original: GraphRNN (You et al., 2018b) with RL.
Revised: GraphRNN (You et al., 2018b) with RL.

Original: MolDQN (Zhou et al., 2019) is based on DQN with three types of actions, Atom addition, Bond addtion, and Bond removal.
Revised: MolDQN (Zhou et al., 2019) is based on DQN with three types of actions, Atom addition, Bond addition, and Bond removal.

Original: The above mentioned work do not use MCTS.
Revised: The above mentioned work do not use MCTS.

Original: Jin et al.
Revised: Jin et al.

Original: (2020) applied MCTS for molecule design with multi-objective constraints, where MCTS is ﬁrst used for extracting important substructures, then a graph-based molecule generation is trained on a data set having these substructures, and outperforms RL based methods.
Revised: (2020) applied MCTS for molecule design with multi-objective constraints, where MCTS is ﬁrst used for extracting important substructures, then a graph-based molecule generation is trained on a data set having these substructures, and outperforms RL based methods.

Original: Yang et al.
Revised: Yang et al.

Original: (2017) combined non-parallel MCTS with a simple GRU and outperformed BO based methods in penalized logP. Sumita et al.
Revised: (2017) combined non-parallel MCTS with a simple GRU and outperformed BO based methods in penalized logP. Sumita et al.

Original: (2018) later applied the same approach to a wavelength problem using a quantum mechanical simulation method (DFT).
Revised: (2018) later applied the same approach to a wavelength problem using a quantum mechanical simulation method (DFT).

Original: 6. P ARALLEL MCTS
Revised: 6. P ARALLEL MCTS

Original: Among the work combining DNN and MCTS beyond single-machine, Distributed AlphaGo (using 1202 GPUs) is the only massively parallel work (applied to Go program).
Revised: Among the work combining DNN and MCTS beyond single-machine, Distributed AlphaGo (using 1202 GPUs) is the only massively parallel work (applied to Go program).

Original: However, it has limited scalability because only one machine holds the search tree, and the GPUs in other machines are used only for neural network computation.
Revised: However, it has limited scalability because only one machine holds the search tree, and the GPUs in other machines are used only for neural network computation.

Original: Hash driven parallel search was ﬁrst applied to distributed MCTS by TDS-df-UCT (Yoshizoe et al., 2011) (for an artiﬁcial game) and UCT-tree-split (Graf et al., 2011) (for the game of Go) independently, and solves communication contention problem.
Revised: Hash driven parallel search was ﬁrst applied to distributed MCTS by TDS-df-UCT (Yoshizoe et al., 2011) (for an artiﬁcial game) and UCT-tree-split (Graf et al., 2011) (for the game of Go) independently, and solves communication contention problem.

Original: UCT-tree-split preserves a replicated subtree for each worker which contains the nodes with large number of visits.
Revised: UCT-tree-split preserves a replicated subtree for each worker which contains the nodes with large number of visits.

Original: It periodically synchronizes the subtrees, hence, suffers from higher communication overhead than TDS-df-UCT.
Revised: It periodically synchronizes the subtrees, hence, suffers from higher communication overhead than TDS-df-UCT.

Original: Other distributed parallel MCTS techniques include Root parallelization and Leaf parallelization.
Revised: Other distributed parallel MCTS techniques include Root parallelization and Leaf parallelization.

Original: Root parallelization performs multiple runs of MCTS with different random seeds (Chaslot et al., 2008) and synchronizes the shallow part of the tree.
Revised: Root parallelization performs multiple runs of MCTS with different random seeds (Chaslot et al., 2008) and synchronizes the shallow part of the tree.

Original: The strength improvement is limited because the size of the tree remains similar to that of non-parallel MCTS (Soejima et al., 2010).
Revised: The strength improvement is limited because the size of the tree remains similar to that of non-parallel MCTS (Soejima et al., 2010).

Original: Another known parallel method is Leaf parallelization (Cazenave & Jouandeau, 2007), but its performance was lower than that of root parallelization.
Revised: Another known parallel method is Leaf parallelization (Cazenave & Jouandeau, 2007), but its performance was lower than that of root parallelization.

Original: Among shared-memory environment (having only a few tens of workers), the ﬁrst reported parallel
Revised: Among shared-memory environment (having only a few tens of workers), the ﬁrst reported parallel

Original: UCT (Gelly et al., 2006), did not rely on virtual loss and it was less efﬁcient.
Revised: UCT (Gelly et al., 2006), did not rely on virtual loss and it was less efﬁcient.

Original: Many other work on shared memory environment use virtual loss (Chaslot et al., 2008; Enzenberger & Müller, 2010;
Revised: Many other work on shared memory environment use virtual loss (Chaslot et al., 2008; Enzenberger & Müller, 2010;

Original: Segal, 2010; Liu et al., 2020).
Revised: Segal, 2010; Liu et al., 2020).

Original: However, they are limited to a maximum of 32 workers.
Revised: However, they are limited to a maximum of 32 workers.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Applying MCTS to molecular design is relatively less explored.
Revised: Applying MCTS to molecular design is relatively less explored.

Original: Ours is the ﬁrst work to explore distributed parallel MCTS for molecular design.
Revised: Ours is the ﬁrst work to explore distributed parallel MCTS for molecular design.

Original: The extensive experiments have shown that an efﬁcient distributed MCTS signiﬁcantly outperforms other approaches that use more complex DNN models combined with optimizations such as Bayesian Optimization or Reinforcement Learning (other than UCT).
Revised: The extensive experiments have shown that an efﬁcient distributed MCTS signiﬁcantly outperforms other approaches that use more complex DNN models combined with optimizations such as Bayesian Optimization or Reinforcement Learning (other than UCT).

Original: Further, it does not trade off search ability (w.r.t non-parallel MCTS) for a real world problem.
Revised: Further, it does not trade off search ability (w.r.t non-parallel MCTS) for a real world problem.

Original: It would be an interesting future work to further enhance the performance of molecular design by using more complex and improved models with parallel MCTS.
Revised: It would be an interesting future work to further enhance the performance of molecular design by using more complex and improved models with parallel MCTS.

Original: Also, MP-MCTS could be applied to other MCTS applications, including retrosynthetic analysis to which an AlphaGo-like approach is applied (Segler et al., 2018b).
Revised: Also, MP-MCTS could be applied to other MCTS applications, including retrosynthetic analysis to which an AlphaGo-like approach is applied (Segler et al., 2018b).

Original: Furthermore, although we analyzed the performance for molecular design, the parallelization technique is independent of the chemistry speciﬁc components and could be applied to other game and non-game applications.
Revised: Furthermore, although we analyzed the performance for molecular design, the parallelization technique is independent of the chemistry speciﬁc components and could be applied to other game and non-game applications.

Original: Our experiments strongly suggest that MCTS can be a better alternative for real-world optimization problems.
Revised: Our experiments strongly suggest that MCTS can be a better alternative for real-world optimization problems.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work is partially supported by JSPS Kakenhi (20H04251).
Revised: This work is partially supported by JSPS Kakenhi (20H04251).

Original: Experiments were conducted on the the RAIDEN computer system at RIKEN AIP.
Revised: Experiments were conducted on the the RAIDEN computer system at RIKEN AIP.

Original: A O THER DETAILS OF PARALLEL MCTS A. S TRENGTH S PEEDUP
Revised: A O THER DETAILS OF PARALLEL MCTS A. S TRENGTH S PEEDUP

Original: It is quite misleading to compare only the number of rollout for evaluating the parallel MCTS algorithms.
Revised: It is quite misleading to compare only the number of rollout for evaluating the parallel MCTS algorithms.

Original: For example, it is easy to achieve a linear speedup in terms of the number of rollouts using the simplistic root parallel MCTS.
Revised: For example, it is easy to achieve a linear speedup in terms of the number of rollouts using the simplistic root parallel MCTS.

Original: However, it is not meaningful because it does not contribute to the actual strength.
Revised: However, it is not meaningful because it does not contribute to the actual strength.

Original: For example, the only massively parallel Go program based on DNN and MCTS, distributed AlphaGo, wins 77% of the games against single-machine AlphaGo using approximately 150-fold more resources (1202 GPUs vs. 8 GPUs)
Revised: For example, the only massively parallel Go program based on DNN and MCTS, distributed AlphaGo, wins 77% of the games against single-machine AlphaGo using approximately 150-fold more resources (1202 GPUs vs. 8 GPUs)

Original: (Silver et al., 2016).
Revised: (Silver et al., 2016).

Original: Distributed AlphaGo is presumed to have limited scalability because only one machine holds the search tree, and the GPUs in other machines are used only for neural network computation.
Revised: Distributed AlphaGo is presumed to have limited scalability because only one machine holds the search tree, and the GPUs in other machines are used only for neural network computation.

Original: But how efﬁcient is this in reality?
Revised: But how efﬁcient is this in reality?

Original: How should we estimate the effective speedup of the distributed version?
Revised: How should we estimate the effective speedup of the distributed version?

Original: In the game-AI domain, strength speedup is used to evaluate the scalability.
Revised: In the game-AI domain, strength speedup is used to evaluate the scalability.

Original: First, measure the strength improvement of the non-parallel program, by using N -times more thinking time than the default settings.
Revised: First, measure the strength improvement of the non-parallel program, by using N -times more thinking time than the default settings.

Original: Then, if the parallel version reaches the same strength within the same default thinking time, using N -times more computing resources, the speedup is ideal.
Revised: Then, if the parallel version reaches the same strength within the same default thinking time, using N -times more computing resources, the speedup is ideal.

Original: Strength speedup was never measured for distributed AlphaGo, but we can guess from the results of other similarly strong Go programs.
Revised: Strength speedup was never measured for distributed AlphaGo, but we can guess from the results of other similarly strong Go programs.

Original: Based on the self-play experiments of ELF OpenGo (Fig.
Revised: Based on the self-play experiments of ELF OpenGo (Fig.

Original: 7, Tian et al.
Revised: 7, Tian et al.

Original: , it is enough to use four times more thinking time to achieve 77% win rate against itself.
Revised: , it is enough to use four times more thinking time to achieve 77% win rate against itself.

Original: Therefore, at least for ELF OpenGo, 77% win rate means less than or equals to 4-fold strength speedup.
Revised: Therefore, at least for ELF OpenGo, 77% win rate means less than or equals to 4-fold strength speedup.

Original: Based on this estimation, we presume that distributed AlphaGo is not efﬁcient.
Revised: Based on this estimation, we presume that distributed AlphaGo is not efﬁcient.

Original: (It is also guessed that distributed AlphaGo was designed to be robust against hardware failures and the scalability was not the ﬁrst priority.)
Revised: (It is also guessed that distributed AlphaGo was designed to be robust against hardware failures and the scalability was not the ﬁrst priority.)

Original: It is difﬁcult to directly compare Go and molecular design, but our "strength speedup" shows almost ideal speedup, obtaining better results (within error range) than non-parallel MCTS (using 256 × more time) by using 256 cores in parallel.
Revised: It is difﬁcult to directly compare Go and molecular design, but our "strength speedup" shows almost ideal speedup, obtaining better results (within error range) than non-parallel MCTS (using 256 × more time) by using 256 cores in parallel.

Original: A. E XTRA M EMORY USAGE OF MP-MCTS
Revised: A. E XTRA M EMORY USAGE OF MP-MCTS

Original: In MP-MCTS, the additional memory consumption per worker of storing w, v, t of siblings at depth d is: C ×
Revised: In MP-MCTS, the additional memory consumption per worker of storing w, v, t of siblings at depth d is: C ×

Original: d × b × n × t , where C = 24 is a constant representing the memory usage of storing w, v, t (each costs 8 bytes) in a node, d is the depth of the tree, b is the average branching factor, n is the number of stored nodes per worker per second, and t is the execution time(s).
Revised: d × b × n × t , where C = 24 is a constant representing the memory usage of storing w, v, t (each costs 8 bytes) in a node, d is the depth of the tree, b is the average branching factor, n is the number of stored nodes per worker per second, and t is the execution time(s).

Original: In our molecular design case, b ≈ 2 . 7 , n ≈ 1 . 8 (one evaluation on a node costs 0.5s, on average).
Revised: In our molecular design case, b ≈ 2 . 7 , n ≈ 1 . 8 (one evaluation on a node costs 0.5s, on average).

Original: Assuming d ≈ 20 , the additional memory usage is 2.3KB per worker per second.
Revised: Assuming d ≈ 20 , the additional memory usage is 2.3KB per worker per second.

Original: For other problems (e.g., AlphaGO), presuming b ≈ 9 , d ≈ 20 , and n ≈ 200 (approximated by one evaluation of a node using DNN costs 5ms), then the additional memory usage is 843KB per worker per second, which is acceptable for recent servers having 10+GB memory per core (or worker) (allows more than 10,000 seconds of thinking time).
Revised: For other problems (e.g., AlphaGO), presuming b ≈ 9 , d ≈ 20 , and n ≈ 200 (approximated by one evaluation of a node using DNN costs 5ms), then the additional memory usage is 843KB per worker per second, which is acceptable for recent servers having 10+GB memory per core (or worker) (allows more than 10,000 seconds of thinking time).

Original: In case the memory usage increases because of the larger b , we can save memory by recording only the top- K branches and ignore less promising branches.
Revised: In case the memory usage increases because of the larger b , we can save memory by recording only the top- K branches and ignore less promising branches.

Original: B V IRTUAL L OSS D ETAILS B. T HE O RIGIN OF V IRTUAL L OSS
Revised: B V IRTUAL L OSS D ETAILS B. T HE O RIGIN OF V IRTUAL L OSS

Original: One of the ﬁrst detailed paper about parallel MCTS (Chaslot et al.
Revised: One of the ﬁrst detailed paper about parallel MCTS (Chaslot et al.

Original: 2008 (Chaslot et al., 2008)) already describes the use of virtual loss (page 5, section 3.3 bottom paragraph titled “Virtual loss”).
Revised: 2008 (Chaslot et al., 2008)) already describes the use of virtual loss (page 5, section 3.3 bottom paragraph titled “Virtual loss”).

Original: The explained behavior about avoiding multiple threads visiting the same leaf is the same as we described in Section 2.3 of the main text.
Revised: The explained behavior about avoiding multiple threads visiting the same leaf is the same as we described in Section 2.3 of the main text.

Original: Yoshizoe et al.
Revised: Yoshizoe et al.

Original: 2011 (Yoshizoe et al., 2011) explains virtual loss in more detail.
Revised: 2011 (Yoshizoe et al., 2011) explains virtual loss in more detail.

Original: Chaslot et al.
Revised: Chaslot et al.

Original: mentions that Coulom 2 suggested, in personal communication, to assign one virtual loss per one thread during phase 1 (e.g. the selection step).
Revised: mentions that Coulom 2 suggested, in personal communication, to assign one virtual loss per one thread during phase 1 (e.g. the selection step).

Original: However, when we personally contacted
Revised: However, when we personally contacted

Original: Coulom to conﬁrm the inventor of virtual loss, it turned out that Coulom was not the inventor.
Revised: Coulom to conﬁrm the inventor of virtual loss, it turned out that Coulom was not the inventor.

Original: We could not track further and the original inventor of virtual loss is still unknown.
Revised: We could not track further and the original inventor of virtual loss is still unknown.

Original: Recently, in early 2020, Liu et al. proposed a method very similar to the vanilla virtual loss (eq. 1) named Watched the Unobserved in UCT (WU-UCT)
Revised: Recently, in early 2020, Liu et al. proposed a method very similar to the vanilla virtual loss (eq. 1) named Watched the Unobserved in UCT (WU-UCT)

Original: (Liu et al., 2020), (eq. 2), We presume Liu et al.
Revised: (Liu et al., 2020), (eq. 2), We presume Liu et al.

Original: misunderstood or overlooked the explanation of virtual loss formula in existing work for some reasons (although they refer to Chaslot et al.
Revised: misunderstood or overlooked the explanation of virtual loss formula in existing work for some reasons (although they refer to Chaslot et al.

Original: (Chaslot et al., 2008), mentioned above).
Revised: (Chaslot et al., 2008), mentioned above).

Original: The explanation about virtual loss in the related work section in (Liu et al.
Revised: The explanation about virtual loss in the related work section in (Liu et al.

Original: (Liu et al., 2020)) and also their comment at the
Revised: (Liu et al., 2020)) and also their comment at the

Original: OpenReview website 3 are wrong.
Revised: OpenReview website 3 are wrong.

Original: In Response to reviewer #4 (part 1 of 2) they say, “To our best knowledge, NONE of the existing TreeP algorithms (or any existing parallel MCTS algorithm) updates the visit counts BEFORE the simulation step ﬁnishes.
Revised: In Response to reviewer #4 (part 1 of 2) they say, “To our best knowledge, NONE of the existing TreeP algorithms (or any existing parallel MCTS algorithm) updates the visit counts BEFORE the simulation step ﬁnishes.

Original: TreeP only updates the values ahead of time using virtual loss.
Revised: TreeP only updates the values ahead of time using virtual loss.

Original: This is also the case for the work [1] and [2].”
Revised: This is also the case for the work [1] and [2].”

Original: However, this is not true.
Revised: However, this is not true.

Original: To our best knowledge, most of the existing work on multithreaded parallel
Revised: To our best knowledge, most of the existing work on multithreaded parallel

Original: MCTS published after 2008 updates the values BEFORE the simulation step completes.
Revised: MCTS published after 2008 updates the values BEFORE the simulation step completes.

Original: (Actually, we can not ﬁnd the beneﬁt of adding virtual loss AFTER the simulation step.)
Revised: (Actually, we can not ﬁnd the beneﬁt of adding virtual loss AFTER the simulation step.)

Original: Liu et al.
Revised: Liu et al.

Original: say AlphaGo (referred as [1] and [2] in their comment) do not update virtual loss BEFORE the simulation step completes.
Revised: say AlphaGo (referred as [1] and [2] in their comment) do not update virtual loss BEFORE the simulation step completes.

Original: However this is presumed to be a misunderstanding caused by the explanation in the ﬁrst AlphaGo paper (Silver et al., 2016).
Revised: However this is presumed to be a misunderstanding caused by the explanation in the ﬁrst AlphaGo paper (Silver et al., 2016).

Original: The paper explains that virtual loss value is added during the Backup phase, so it sounds like the value is added after the simulation (explained in Methods, Search Algorithms Section, in the paper (Silver et al., 2016)).
Revised: The paper explains that virtual loss value is added during the Backup phase, so it sounds like the value is added after the simulation (explained in Methods, Search Algorithms Section, in the paper (Silver et al., 2016)).

Original: However, if you read carefully, the paper says that the virtual loss is added before the end of the simulation, and removed (subtracted) after the end of the simulation.
Revised: However, if you read carefully, the paper says that the virtual loss is added before the end of the simulation, and removed (subtracted) after the end of the simulation.

Original: Therefore, AlphaGo does update the visit count before the simulation step ﬁnishes.
Revised: Therefore, AlphaGo does update the visit count before the simulation step ﬁnishes.

Original: The main difference between WU-UCT and existing Virtual-loss-based parallel MCTS is the difference of the two formulas (eq. 1 and 2).
Revised: The main difference between WU-UCT and existing Virtual-loss-based parallel MCTS is the difference of the two formulas (eq. 1 and 2).

Original: [Equation]
Revised: [Equation]

Original: B. C OMPARISON OF V IRTUAL L OSS F ORMULAS
Revised: B. C OMPARISON OF V IRTUAL L OSS F ORMULAS

Original: We compared the results of our MP-MCTS using three different virtual loss formulas, the vanilla virtual loss, WU (Liu et al., 2020), and a new formula shown in eq.
Revised: We compared the results of our MP-MCTS using three different virtual loss formulas, the vanilla virtual loss, WU (Liu et al., 2020), and a new formula shown in eq.

Original: 3, UCB vl LCB (LCB stands for
Revised: 3, UCB vl LCB (LCB stands for

Original: Lower Conﬁdence Bound).
Revised: Lower Conﬁdence Bound).

Original: Vanilla virtual loss assumes zero reward from the ongoing simulations and WU assumes the reward remains unchanged.
Revised: Vanilla virtual loss assumes zero reward from the ongoing simulations and WU assumes the reward remains unchanged.

Original: UCB vl LCB assumes something between these two, assumes a decreased reward estimated by Lower Conﬁdence Bound.
Revised: UCB vl LCB assumes something between these two, assumes a decreased reward estimated by Lower Conﬁdence Bound.

Original: Table 3 shows the logP score results of our MP-MCTS using three virtual loss formulas, averaged for 10 runs.
Revised: Table 3 shows the logP score results of our MP-MCTS using three virtual loss formulas, averaged for 10 runs.

Original: The results suggests that, for molecular design, WU and UCB vl LCB do not improve the results over vanilla virtual loss.
Revised: The results suggests that, for molecular design, WU and UCB vl LCB do not improve the results over vanilla virtual loss.

Original: [Table]
Revised: [Table]

Original: B. E XAMPLES OF V IRTUAL L OSS IN C ODES
Revised: B. E XAMPLES OF V IRTUAL L OSS IN C ODES

Original: It is common to use virtual loss for parallel game programs.
Revised: It is common to use virtual loss for parallel game programs.

Original: We show real examples of virtual loss implementations in existing open source Go or other game programs, both before and after AlphaGo.
Revised: We show real examples of virtual loss implementations in existing open source Go or other game programs, both before and after AlphaGo.

Original: We can see the examples of the usage of virtual loss, and in all of these, the values are updated
Revised: We can see the examples of the usage of virtual loss, and in all of these, the values are updated

Original: BEFORE the simulation step completes.
Revised: BEFORE the simulation step completes.

Original: It is also interesting to note that the majority of these work modify virtual loss equation for their implementations because they rely on different variations of
Revised: It is also interesting to note that the majority of these work modify virtual loss equation for their implementations because they rely on different variations of

Original: UCT, such as P-UCT (Rosin, 2010).
Revised: UCT, such as P-UCT (Rosin, 2010).

Original: Fuego:
Revised: Fuego:

Original: Fuego (Enzenberger et al., 2010), one of the best open source Go programs until 2015, is a generic game library.
Revised: Fuego (Enzenberger et al., 2010), one of the best open source Go programs until 2015, is a generic game library.

Original: It started to use virtual loss in 2008 (from r677, committed on Nov. 27, 2008).
Revised: It started to use virtual loss in 2008 (from r677, committed on Nov. 27, 2008).

Original: In the PlayGame function starting from line 673 (in the following URL), it calls AddVirtualLoss at line 685, clearly before the playout (simulation), playouts start right below at line 693 after
Revised: In the PlayGame function starting from line 673 (in the following URL), it calls AddVirtualLoss at line 685, clearly before the playout (simulation), playouts start right below at line 693 after

Original: StartPlayouts .
Revised: StartPlayouts .

Original: https://sourceforge.net/p/fuego/code/HEAD/tree/tags/VERSION_0_3/ smartgame/SgUctSearch.cpp
Revised: https://sourceforge.net/p/fuego/code/HEAD/tree/tags/VERSION_0_3/ smartgame/SgUctSearch.cpp

Original: ELF OpenGo:
Revised: ELF OpenGo:

Original: It is a Go program and generic game library developed by FaceBook researchers (Tian et al., 2019).
Revised: It is a Go program and generic game library developed by FaceBook researchers (Tian et al., 2019).

Original: Source code of the search part is in the following URL.
Revised: Source code of the search part is in the following URL.

Original: In the single_rollout function starting from line 258, it calls addVirtualLoss at line 282.
Revised: In the single_rollout function starting from line 258, it calls addVirtualLoss at line 282.

Original: (Please note that here “rollout” means the one whole cycle of the UCT, start selection from the root, reach a leaf, do a (random
Revised: (Please note that here “rollout” means the one whole cycle of the UCT, start selection from the root, reach a leaf, do a (random

Original: ) rollout
Revised: ) rollout

Original: , and
Revised: , and

Original: backpropagate
Revised: backpropagate

Original: https://github.com/pytorch/ELF/blob/113aba73ec0bc9d60bdb00b3c439bc60fecabc89/
Revised: https://github.com/pytorch/ELF/blob/113aba73ec0bc9d60bdb00b3c439bc60fecabc89/

Original: src_cpp/elf/ai/tree_search/tree_search.h
Revised: src_cpp/elf/ai/tree_search/tree_search.h

Original: LeelaZero:
Revised: LeelaZero:

Original: An open source Go program.
Revised: An open source Go program.

Original: In play_simulation function starting from line 59, virtual_loss updates the values Please note that this is a part of the selection step.
Revised: In play_simulation function starting from line 59, virtual_loss updates the values Please note that this is a part of the selection step.

Original: play_simulation is recursively called at line 88 or 94.
Revised: play_simulation is recursively called at line 88 or 94.

Original: https://github.com/leela-zero/leela-zero/blob/0d1791e3f4de1f52389fe41d341484f4f66ea1e9/ src/UCTSearch.cpp
Revised: https://github.com/leela-zero/leela-zero/blob/0d1791e3f4de1f52389fe41d341484f4f66ea1e9/ src/UCTSearch.cpp

Original: AQ: An open source Go program.
Revised: AQ: An open source Go program.

Original: Virtual loss is added before simulations at line 142, and subtracted at line 199 or around line 247 after simulations.
Revised: Virtual loss is added before simulations at line 142, and subtracted at line 199 or around line 247 after simulations.

Original: Also it is interesting to note that AQ uses virtual loss in two different ways, one for random rollouts and one for Neural Network based evaluation.
Revised: Also it is interesting to note that AQ uses virtual loss in two different ways, one for random rollouts and one for Neural Network based evaluation.

Original: https://github.
Revised: https://github.

Original: com
Revised: com

Original: ymgaq
Revised: ymgaq

Original: /AQ/blob/36f6728f2f817c2fb0c69d73b00ce155582edb10/ src/search.cc
Revised: /AQ/blob/36f6728f2f817c2fb0c69d73b00ce155582edb10/ src/search.cc

Original: C T OP M OLECULES C. LOG P OPTIMIZATION
Revised: C T OP M OLECULES C. LOG P OPTIMIZATION

Original: Figure 1 (a) shows the top 3 molecules by MP-MCTS for penalized logP optimization.
Revised: Figure 1 (a) shows the top 3 molecules by MP-MCTS for penalized logP optimization.

Original: MP-MCTS can design molecules with extremely high penalized logP scores, which demonstrates that our MP-MCTS algorithm has the ability of identifying the promising branches.
Revised: MP-MCTS can design molecules with extremely high penalized logP scores, which demonstrates that our MP-MCTS algorithm has the ability of identifying the promising branches.

Original: However, these molecules are not desirable because they are likely to have inaccurate predicted properties, which shows an limitation of maximizing penalized logP using an empirical tool (e.g. RDKit).
Revised: However, these molecules are not desirable because they are likely to have inaccurate predicted properties, which shows an limitation of maximizing penalized logP using an empirical tool (e.g. RDKit).

Original: Therefore it would be interesting to apply MCTS based approach for a different optimization problem with more accurate simulations.
Revised: Therefore it would be interesting to apply MCTS based approach for a different optimization problem with more accurate simulations.

Original: Figure 1 (b) shows the best three molecules designed by MP-MCTS for another problem, wavelength property optimization (explained below).
Revised: Figure 1 (b) shows the best three molecules designed by MP-MCTS for another problem, wavelength property optimization (explained below).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: W AVELENGTH OPTIMIZATION
Revised: W AVELENGTH OPTIMIZATION

Original: It is possible to predict the wavelength of a given molecule using quantum mechanical simulation methods based on Density-Functional Theory (DFT).
Revised: It is possible to predict the wavelength of a given molecule using quantum mechanical simulation methods based on Density-Functional Theory (DFT).

Original: Following Sumita et al.
Revised: Following Sumita et al.

Original: (Sumita et al., 2018)
Revised: (Sumita et al., 2018)

Original: [Table]
Revised: [Table]

Original: we apply our MP-MCTS for ﬁnding molecules with greater absorption wavelength but limited to 2500 nm.
Revised: we apply our MP-MCTS for ﬁnding molecules with greater absorption wavelength but limited to 2500 nm.

Original: Model and dataset.
Revised: Model and dataset.

Original: Our model is based on GRU, which mainly consists of one GRU layer.
Revised: Our model is based on GRU, which mainly consists of one GRU layer.

Original: Input data represents 27 SMILES symbols in one-hot vectors, which represents the symbols appeared in the training dataset.
Revised: Input data represents 27 SMILES symbols in one-hot vectors, which represents the symbols appeared in the training dataset.

Original: The GRU layer has 27-dim input/256-dim output and the last dense layer has outputs 27 values with softmax.
Revised: The GRU layer has 27-dim input/256-dim output and the last dense layer has outputs 27 values with softmax.

Original: The model was pre-trained using a molecule dataset that contains 13K molecules (only H, O, N, and C elements included) extracted from the PubChemQC database (Nakata & Shimazaki, 2017).
Revised: The model was pre-trained using a molecule dataset that contains 13K molecules (only H, O, N, and C elements included) extracted from the PubChemQC database (Nakata & Shimazaki, 2017).

Original: Calculating wavelength property and reward function.
Revised: Calculating wavelength property and reward function.

Original: Following (Sumita et al., 2018), the wavelength property was evaluated by DFT calculations with B3LYP/3-21G ∗ level setting using
Revised: Following (Sumita et al., 2018), the wavelength property was evaluated by DFT calculations with B3LYP/3-21G ∗ level setting using

Original: Gaussian 16 (Frisch et al., 2016).
Revised: Gaussian 16 (Frisch et al., 2016).

Original: We used one core (for simplicity) for the DFT calculation.
Revised: We used one core (for simplicity) for the DFT calculation.

Original: The reward function is deﬁned as equation 4.
Revised: The reward function is deﬁned as equation 4.

Original: If the generated molecules are valid, then we assign a positive reward within ( − 1 , 1] as shown in equation 4.
Revised: If the generated molecules are valid, then we assign a positive reward within ( − 1 , 1] as shown in equation 4.

Original: Negative reward -1 is assigned in case the generated molecule is invalid, DFT fails, or the wavelength is greater than the limit.
Revised: Negative reward -1 is assigned in case the generated molecule is invalid, DFT fails, or the wavelength is greater than the limit.

Original: [Equation]
Revised: [Equation]

Original: Optimization of wavelength property.
Revised: Optimization of wavelength property.

Original: Table 2 summarizes the optimized wavelength (nm) of
Revised: Table 2 summarizes the optimized wavelength (nm) of

Original: MP-MCTS approach for varying number of CPU cores.
Revised: MP-MCTS approach for varying number of CPU cores.

Original: 3 independent runs and 6 hours time limit were applied to each setting, and the average and standard deviation of the best scores were shown.
Revised: 3 independent runs and 6 hours time limit were applied to each setting, and the average and standard deviation of the best scores were shown.

Original: C. S IMILARITY S EARCH
Revised: C. S IMILARITY S EARCH

Original: To test our MP-MCTS for another type of problem, we ran our algorithm using Zaleplon MPO, one of the Guacamol benchmarks (Brown et al., 2019).
Revised: To test our MP-MCTS for another type of problem, we ran our algorithm using Zaleplon MPO, one of the Guacamol benchmarks (Brown et al., 2019).

Original: Unlike the property based scores (e.g., penalized logP or the wavelength score), Zaleplon MPO aims to ﬁnd similar
Revised: Unlike the property based scores (e.g., penalized logP or the wavelength score), Zaleplon MPO aims to ﬁnd similar

Original: but not exactly the same molecules as the given target molecule.
Revised: but not exactly the same molecules as the given target molecule.

Original: Table 5 shows the results for 10 minutes x 5 runs.
Revised: Table 5 shows the results for 10 minutes x 5 runs.

Original: The best score was 0.659, which we observed in one of the runs for 256 cores.
Revised: The best score was 0.659, which we observed in one of the runs for 256 cores.

Original: The score is lower than the best score of 0.754 achieved by Graph GA, according to the Guacamol website.
Revised: The score is lower than the best score of 0.754 achieved by Graph GA, according to the Guacamol website.

Original: Further validation is needed to assess the effectiveness of our method for this type of problem.
Revised: Further validation is needed to assess the effectiveness of our method for this type of problem.

Original: O N + ONH
Revised: O N + ONH

Original: O
Revised: O

Original: N N
Revised: N N

Original: O
Revised: O

Original: F
Revised: F

Original: O N + ONH
Revised: O N + ONH

Original: O
Revised: O

Original: N N
Revised: N N

Original: O
Revised: O

Original: Cl
Revised: Cl

Original: O N + ONH
Revised: O N + ONH

Original: O
Revised: O

Original: N N
Revised: N N

Original: O Cl 15.
Revised: O Cl 15.

Original: O=[N+]([O-])c1ccc2[nH]ccc2c1OCc1cncnc1Oc1cccc(C#CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCl)c
Revised: O=[N+]([O-])c1ccc2[nH]ccc2c1OCc1cncnc1Oc1cccc(C#CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCl)c

Original: O=[N+]([O-])c1ccc2[nH]ccc2c1OCc1cncnc1Oc1cccc(C#CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCl)c O=[N+]([O-])c1ccc2[nH]ccc2c1OCc1cncnc1Oc1cccc(C#CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCF)c
Revised: O=[N+]([O-])c1ccc2[nH]ccc2c1OCc1cncnc1Oc1cccc(C#CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCl)c O=[N+]([O-])c1ccc2[nH]ccc2c1OCc1cncnc1Oc1cccc(C#CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCF)c

Original: N O
Revised: N O

Original: N
Revised: N

Original: OH
Revised: OH

Original: NH
Revised: NH

Original: OH
Revised: OH

Original: O
Revised: O

Original: N N N
Revised: N N N

Original: O
Revised: O

Original: N
Revised: N

Original: O
Revised: O

Original: N N
Revised: N N

Original: O
Revised: O

Original: OH 2374.6 nm 2447.2 nm 2455.4 nm
Revised: OH 2374.6 nm 2447.2 nm 2455.4 nm

Original: Cc1cc(no1)[C](/N=C(O)/[NH])O
Revised: Cc1cc(no1)[C](/N=C(O)/[NH])O

Original: O=Nc1nccnc1C(=O)N(C)C O=Nc1ncccc1C(=O)O
Revised: O=Nc1nccnc1C(=O)N(C)C O=Nc1ncccc1C(=O)O

Original: [Figure]
Revised: [Figure]

Original: D P SEUDO CODE
Revised: D P SEUDO CODE

Original: The Pseudocode for TDS-UCT, TDS-df-UCT, and MP-MCTS are shown in Algorithm 1, 2, and 3.
Revised: The Pseudocode for TDS-UCT, TDS-df-UCT, and MP-MCTS are shown in Algorithm 1, 2, and 3.

Original: All of the workers call the function TDS_UCT(), TDS_df_UCT(), or MP_MCTS() respectively, and continue until timeup.
Revised: All of the workers call the function TDS_UCT(), TDS_df_UCT(), or MP_MCTS() respectively, and continue until timeup.

Original: 1: function TDS_UCT() 2:
Revised: 1: function TDS_UCT() 2:

Original: Initialize Hash_Table, Initialize job_queue, N_jobs=3 ×
Revised: Initialize Hash_Table, Initialize job_queue, N_jobs=3 ×

Original: #workers, 3: initial_job =
Revised: #workers, 3: initial_job =

Original: [SELECT, root_node] 4: if this worker is the home_worker(root_node) then 5: for N_jobs do 6: job_queue.Push(initial_job) 7: end for 8: end if 9: while TimeIsRemaining do 10:
Revised: [SELECT, root_node] 4: if this worker is the home_worker(root_node) then 5: for N_jobs do 6: job_queue.Push(initial_job) 7: end for 8: end if 9: while TimeIsRemaining do 10:

Original: Receive all incoming messages and push to job_queue 11: if job_queue.
Revised: Receive all incoming messages and push to job_queue 11: if job_queue.

Original: NotEmpty() then 12: (job_type, node) = job_queue.
Revised: NotEmpty() then 12: (job_type, node) = job_queue.

Original: Pop() 13: if job_type ==
Revised: Pop() 13: if job_type ==

Original: SELECT then 14: if node is in HashTable then 15: node = LookUpHashTable(node) 16: if node not yet expanded then Expansion(node) end if 17: (cid:46)
Revised: SELECT then 14: if node is in HashTable then 15: node = LookUpHashTable(node) 16: if node not yet expanded then Expansion(node) end if 17: (cid:46)

Original: Expand node.children on the second visit 18: best_child=Selection(node) 19:
Revised: Expand node.children on the second visit 18: best_child=Selection(node) 19:

Original: AddVirtualLoss(node.children[best_child]
Revised: AddVirtualLoss(node.children[best_child]

Original: WriteToHashTable(node) 21:
Revised: WriteToHashTable(node) 21:

Original: SendMessage([SELECT, best_child], dest=home_worker(best_child)) 22: else 23:
Revised: SendMessage([SELECT, best_child], dest=home_worker(best_child)) 22: else 23:

Original: Reward=Simulation(node) 24: node.
Revised: Reward=Simulation(node) 24: node.

Original: Update(Reward) 25:
Revised: Update(Reward) 25:

Original: WriteToHashTable(node) 26:
Revised: WriteToHashTable(node) 26:

Original: SendMessage([BP, node], dest=home_worker(node.parent)) 27: end if 28: else if job_type ==
Revised: SendMessage([BP, node], dest=home_worker(node.parent)) 27: end if 28: else if job_type ==

Original: BP then 29: parent = LookUpHashTable(node.parent) (cid:46) parent should be in hashtable on BP 30:
Revised: BP then 29: parent = LookUpHashTable(node.parent) (cid:46) parent should be in hashtable on BP 30:

Original: RemoveVirtualLoss(parent.children[node]) 31: parent.
Revised: RemoveVirtualLoss(parent.children[node]) 31: parent.

Original: Update(node.reward) 32:
Revised: Update(node.reward) 32:

Original: WriteToHashTable(parent) 33: if parent !
Revised: WriteToHashTable(parent) 33: if parent !

Original: =
Revised: =

Original: root_node then 34:
Revised: root_node then 34:

Original: SendMessage([BP, parent], dest=home_worker(parent.parent)) 35: else 36: best_child=Selection(parent) 37:
Revised: SendMessage([BP, parent], dest=home_worker(parent.parent)) 35: else 36: best_child=Selection(parent) 37:

Original: AddVirtualLoss(parent.children[best_child]) 38:
Revised: AddVirtualLoss(parent.children[best_child]) 38:

Original: SendMessage([SELECT, best_child], dest=home_worker(best_child)) 39: end if 40: end if 41: end if 42: end while 43: end function
Revised: SendMessage([SELECT, best_child], dest=home_worker(best_child)) 39: end if 40: end if 41: end if 42: end while 43: end function

Original: 1: function TDS_ DF _UCT() 2:
Revised: 1: function TDS_ DF _UCT() 2:

Original: Initialize Hash_Table, Initialize job_queue, N_jobs=3 ×
Revised: Initialize Hash_Table, Initialize job_queue, N_jobs=3 ×

Original: #workers, 3: initial_job =
Revised: #workers, 3: initial_job =

Original: [SELECT, root_node, None] 4: if this worker is the home_worker(root_node) then 5: for N_jobs do 6: job_queue.Push(initial_job) 7: end for 8: end if 9: while TimeIsRemaining do 10:
Revised: [SELECT, root_node, None] 4: if this worker is the home_worker(root_node) then 5: for N_jobs do 6: job_queue.Push(initial_job) 7: end for 8: end if 9: while TimeIsRemaining do 10:

Original: Receive all incoming messages and push to job_queue 11: if job_queue.
Revised: Receive all incoming messages and push to job_queue 11: if job_queue.

Original: NotEmpty() then 12: (job_type, node, ucb_history) = job_queue.
Revised: NotEmpty() then 12: (job_type, node, ucb_history) = job_queue.

Original: Pop() 13: if job_type ==
Revised: Pop() 13: if job_type ==

Original: SELECT then 14: if node is in HashTable then 15: node = LookUpHashTable(node) 16: if node not yet expanded then Expansion(node) end if 17: (cid:46)
Revised: SELECT then 14: if node is in HashTable then 15: node = LookUpHashTable(node) 16: if node not yet expanded then Expansion(node) end if 17: (cid:46)

Original: Expand node.children on the second visit 18: best_child=Selection(node) 19:
Revised: Expand node.children on the second visit 18: best_child=Selection(node) 19:

Original: AddVirtualLoss(node.children[best_child]
Revised: AddVirtualLoss(node.children[best_child]

Original: WriteToHashTable(node) 21: ucb_history.
Revised: WriteToHashTable(node) 21: ucb_history.

Original: Append(node.children)
Revised: Append(node.children)

Original: (cid:46) ucb_history is only in messages 22:
Revised: (cid:46) ucb_history is only in messages 22:

Original: SendMessage([SELECT, best_child, ucb_history], 23: dest=home_worker(best_child)) 24: else 25:
Revised: SendMessage([SELECT, best_child, ucb_history], 23: dest=home_worker(best_child)) 24: else 25:

Original: Reward=Simulation(node) 26: node.
Revised: Reward=Simulation(node) 26: node.

Original: Update(Reward) 27:
Revised: Update(Reward) 27:

Original: WriteToHashTable(node) 28: ucb_history.
Revised: WriteToHashTable(node) 28: ucb_history.

Original: RemoveBottomRow() (cid:46) ucb_history is only in messages 29:
Revised: RemoveBottomRow() (cid:46) ucb_history is only in messages 29:

Original: SendMessage([BP, node, node.ucb_history], 30: dest=home_worker(node.parent)) 31: end if 32: else if job_type ==
Revised: SendMessage([BP, node, node.ucb_history], 30: dest=home_worker(node.parent)) 31: end if 32: else if job_type ==

Original: BP then 33: parent = LookUpHashTable(node.parent) (cid:46) parent should be in hashtable on BP 34:
Revised: BP then 33: parent = LookUpHashTable(node.parent) (cid:46) parent should be in hashtable on BP 34:

Original: RemoveVirtualLoss(parent.children[node]) 35: parent.
Revised: RemoveVirtualLoss(parent.children[node]) 35: parent.

Original: Update(node.reward) 36:
Revised: Update(node.reward) 36:

Original: WriteToHashTable(parent) 37: current_best_node = ucb_history.
Revised: WriteToHashTable(parent) 37: current_best_node = ucb_history.

Original: GetCurrentBest() 38: ucb_history.
Revised: GetCurrentBest() 38: ucb_history.

Original: RemoveBottomRow() (cid:46) ucb_history is only in messages 39: if parent ==
Revised: RemoveBottomRow() (cid:46) ucb_history is only in messages 39: if parent ==

Original: current_best_node OR parent =
Revised: current_best_node OR parent =

Original: = root_node then 40:
Revised: = root_node then 40:

Original: best_child=Selection(parent) 41:
Revised: best_child=Selection(parent) 41:

Original: AddVirtualLoss(parent.children[best_child]) 42:
Revised: AddVirtualLoss(parent.children[best_child]) 42:

Original: WriteToHashTable(parent) 43: ucb_history.
Revised: WriteToHashTable(parent) 43: ucb_history.

Original: Append(parent.children)
Revised: Append(parent.children)

Original: (cid:46) ucb_history is only in messages 44:
Revised: (cid:46) ucb_history is only in messages 44:

Original: SendMessage([SELECT, best_child, ucb_history], 45: dest=home_worker(best_child)) 46: else 47:
Revised: SendMessage([SELECT, best_child, ucb_history], 45: dest=home_worker(best_child)) 46: else 47:

Original: SendMessage([BP, parent, parent.ucb_history], 48: dest=home_worker(parent.parent)) 49: end if 50: end if 51: end if 52: end while 53: end function
Revised: SendMessage([BP, parent, parent.ucb_history], 48: dest=home_worker(parent.parent)) 49: end if 50: end if 51: end if 52: end while 53: end function

Original: 1: function MP_MCTS() 2:
Revised: 1: function MP_MCTS() 2:

Original: Initialize Hash_Table, Initialize job_queue, N_jobs=3 ×
Revised: Initialize Hash_Table, Initialize job_queue, N_jobs=3 ×

Original: #workers, 3: initial_job =
Revised: #workers, 3: initial_job =

Original: [SELECT, root_node, None] 4: if this worker is the home_worker(root_node) then 5: for N_jobs do 6: job_queue.Push(initial_job) 7: end for 8: end if 9: while TimeIsRemaining do 10:
Revised: [SELECT, root_node, None] 4: if this worker is the home_worker(root_node) then 5: for N_jobs do 6: job_queue.Push(initial_job) 7: end for 8: end if 9: while TimeIsRemaining do 10:

Original: Receive all incoming messages and push to job_queue 11: if job_queue.
Revised: Receive all incoming messages and push to job_queue 11: if job_queue.

Original: NotEmpty() then 12: (job_type, node, ucb_history) = job_queue.
Revised: NotEmpty() then 12: (job_type, node, ucb_history) = job_queue.

Original: Pop() 13: if job_type ==
Revised: Pop() 13: if job_type ==

Original: SELECT then 14: if node is in HashTable then 15: node = LookUpHashTable(node) 16: if node not yet expanded then Expansion(node) end if 17: (cid:46)
Revised: SELECT then 14: if node is in HashTable then 15: node = LookUpHashTable(node) 16: if node not yet expanded then Expansion(node) end if 17: (cid:46)

Original: Expand node.children on the second visit 18: best_child=Selection(node) 19:
Revised: Expand node.children on the second visit 18: best_child=Selection(node) 19:

Original: AddVirtualLoss(node.children[best_child]
Revised: AddVirtualLoss(node.children[best_child]

Original: ) 20: node.
Revised: ) 20: node.

Original: UpdateUCBHistory(ucb_history) 21:
Revised: UpdateUCBHistory(ucb_history) 21:

Original: WriteToHashTable(node) 22: ucb_history.
Revised: WriteToHashTable(node) 22: ucb_history.

Original: Append(node.children) 23:
Revised: Append(node.children) 23:

Original: SendMessage([SELECT, best_child, ucb_history], 24: dest=home_worker(best_child)) 25: else 26:
Revised: SendMessage([SELECT, best_child, ucb_history], 24: dest=home_worker(best_child)) 25: else 26:

Original: Reward=Simulation(node) 27: node.
Revised: Reward=Simulation(node) 27: node.

Original: Update(Reward) 28:
Revised: Update(Reward) 28:

Original: WriteToHashTable(node) 29:
Revised: WriteToHashTable(node) 29:

Original: SendMessage([BP, node, node.ucb_history], 30: dest=home_worker(node.parent)) 31: end if 32: else if job_type ==
Revised: SendMessage([BP, node, node.ucb_history], 30: dest=home_worker(node.parent)) 31: end if 32: else if job_type ==

Original: BP then 33: parent = LookUpHashTable(node.parent) (cid:46) parent should be in hashtable on BP 34:
Revised: BP then 33: parent = LookUpHashTable(node.parent) (cid:46) parent should be in hashtable on BP 34:

Original: RemoveVirtualLoss(parent.children[node]) 35: parent.
Revised: RemoveVirtualLoss(parent.children[node]) 35: parent.

Original: Update(node.reward) 36: parent.
Revised: Update(node.reward) 36: parent.

Original: UpdateUCBHistory(ucb_history) 37:
Revised: UpdateUCBHistory(ucb_history) 37:

Original: WriteToHashTable(parent) 38: current_best_node = parent.ucb_history.
Revised: WriteToHashTable(parent) 38: current_best_node = parent.ucb_history.

Original: GetCurrentBest() 39: if parent == current_best_node OR parent =
Revised: GetCurrentBest() 39: if parent == current_best_node OR parent =

Original: =
Revised: =

Original: root_node then 40:
Revised: root_node then 40:

Original: best_child=Selection(parent) 41:
Revised: best_child=Selection(parent) 41:

Original: AddVirtualLoss(parent.children[best_child]) 42:
Revised: AddVirtualLoss(parent.children[best_child]) 42:

Original: WriteToHashTable(parent) 43: ucb_history.
Revised: WriteToHashTable(parent) 43: ucb_history.

Original: Append(parent.children) 44:
Revised: Append(parent.children) 44:

Original: SendMessage([SELECT, best_child, ucb_history], 45: dest=home_worker(best_child)) 46: else 47:
Revised: SendMessage([SELECT, best_child, ucb_history], 45: dest=home_worker(best_child)) 46: else 47:

Original: SendMessage([BP, parent, parent.ucb_history], 48: dest=home_worker(parent.parent)) 49: end if 50: end if 51: end if 52: end while 53: end function
Revised: SendMessage([BP, parent, parent.ucb_history], 48: dest=home_worker(parent.parent)) 49: end if 50: end if 51: end if 52: end while 53: end function