M ODEL - BASED REINFORCEMENT LEARNING FOR BIOLOGICAL SEQUENCE DESIGN
Anonymous authors
Paper under double-blind review
A BSTRACT
The ability to design biological structures such as DNA or proteins would have considerable medical and industrial impact.
Doing so presents a challenging black-box optimization problem characterized by the large-batch, low round setting due to the need for labor-intensive wet lab evaluations.
In response, we propose using reinforcement learning (RL) based on proximal-policy optimization (PPO) for biological sequence design.
RL provides a ﬂexible framework for optimization generative sequence models to achieve speciﬁc criteria, such as diversity among the high-quality sequences discovered.
We propose a model-based variant of PPO, DyNA-PPO, to improve sample efﬁciency, where the policy for a new round is trained ofﬂine using a simulator ﬁt on functional measurements from prior rounds.
To accommodate the growing number of observations across rounds, the simulator model is automatically selected at each round from a pool of diverse models of varying capacity.
On the tasks of designing DNA transcription factor binding sites, designing antimicrobial proteins, and optimizing the energy of Ising models based on protein structure, we ﬁnd that DyNA-PPO performs signiﬁcantly better than existing methods in settings in which modeling is feasible, while still not performing worse in situations in which a reliable model cannot be learned.
I NTRODUCTION
Driven by real-world obstacles in health and disease requiring new drugs, treatments, and assays, the goal of biomolecular design is to identify new discrete sequences x which optimize some oracle, typically an experimentally-measured functional property f ( x ) .
This is a difﬁcult black-box optimization problem over a combinatorially large search space in which function evaluation relies on slow and expensive wet-lab experiments.
The setting induces unusual constraints in blackbox optimization and reinforcement learning: large synchronous batches with few rounds total.
The current gold standard for biomolecular design is directed evolution , which was recently recognized with a Nobel prize (Arnold, 1998) and is a form of randomized local search.
While the method has had considerable impact, it has low sample efﬁciency and relies on greedy hillclimbing to the optimal sequences.
Recent work has demonstrated that machine-learning-guided optimization (Section 3) can ﬁnd better sequences faster.
Reinforcement learning (RL) provides a ﬂexible framework for black-box optimization that can harness modern deep generative sequence models.
This paper proposes a simple method for improving the sample efﬁciency of policy gradient methods such as PPO (Schulman et al., 2017) for black-box optimization by using surrogate models that are trained online to approximate f ( x ) .
Our method updates the policy’s parameters using sequences x generated by the current policy π θ ( x ) , but evaluated using a learned surrogate f (cid:48) w ( x ) , instead of the true, but unknown, oracle reward function f ( x ) .
We learn the parameters of the reward model, w , simultaneously with the parameters of the policy.
This is similar to other model-based RL methods, but simpler, since in the context of sequence optimization, the state-transition model is deterministic and known.
Initially the learned reward model, f (cid:48)
w ( x ) , is unreliable, so we rely entirely on f ( x ) to assess sequences and update the policy.
This allows a graceful fallback to PPO when the model is not effective.
Over time, the reward model becomes more reliable and can be used as a cheap surrogate, similar to Bayesian optimization methods (Shahriari et al., 2015).
We show empirically that cross-validation is an effective heuristic for assessing the model quality, which is simpler than the inference required by Bayesian optimization.
We rigorously evaluate our method on three in-silico sequence design tasks that draw on experimental data to construct functions f ( x ) characteristic of real-world design problems: optimizing binding afﬁnity of DNA sequences of length 8 (state space has size 4 8 ); optimizing anti-microbial peptide sequences (state space has size 20 50 ), and optimizing binary sequences where f ( x ) is deﬁned by the energy of an Ising model for protein structure (state space has size 2 50 ).
These do not rely on wet lab experiments, and thus allow for large-scale benchmarking across a range of methods.
We show that our DyNA-PPO method achieves higher cumulative reward for a given budget (measured in terms of number of calls to f ( x ) ) than various other methods from the literature, such as standard
PPO, various forms of the cross-entropy method, Bayesian optimization and evolutionary search
In summary, our contributions are as follows:
• We provide a model-based RL algorthm, DyNA-PPO, and demonstrate its effectiveness in the large batch / low round constraints imposed by biological sequence design.
• We demonstrate that model-based RL can increase sample efﬁciency, and we address model bias by quantifying the reliability and automatically selecting models of appropriate complexity via cross validation.
• We propose a visitation-based exploration bonus and show that it is more effective than entropy-regularization in identifying multiple local optima.
• We present a new optimization task for benchmarking methods for biological sequence design based on protein energy Ising models.
M ETHODS
Let f ( x ) be the function that we want to optimize and x ∈ V T a sequence of length T over a vocabulary V such as DNA nucleotides ( | V | = 4) or amino acids ( | V | = 20).
We assume N experimental rounds and that B sequences can be measured per round.
Let D n = { ( x, f ( x )) } be the data acquired in round n with | D n | =
B .
For simplicity, we assume that the sequence length T is constant, but our approach based on generating sequences autoregressively easily generalizes to variable-length sequences.
2. M ARKOV D ECISION P ROCESS
We formulate the design of a single sequence x as a Markov decision process M = ( S, A, p, r ) with state space S , action space A , transition function p , and reward function r .
The state space
S = ∪ t =1 ...T V t is the set of all possible sequence preﬁxes and A corresponds to the vocabulary V .
A sequence is generated left to right.
At time step t , the state s t = a 0 , ..., a t − 1 corresponds to the t last tokens and the action a t ∈
A to the next token.
The transition function p ( s t + 1 | s t )
=
s t a t is deterministic and corresponds to appending a t to s t .
The reward r ( s t , a t ) is zero except at the last step T , where it corresponds to the functional measurement f
( s T − 1 ) .
For generating variable length sequences, we extend the vocabulary by a special end-of-sequence token and terminate sequence generation when this token is selected.
2. P OLICY O PTIMIZATION
We train a policy π θ ( a t | s t ) to optimize the expected sum of rewards :
[Equation]
We build off of proximal policy optimization (PPO) with a KL-penalty (Schulman et al., 2017), which we have found to be more stable and sample efﬁcient than REINFORCE (Williams, 1992).
We have also considered off-policy deep Q-learning (Mnih et al., 2015), categorical distributional deep Q-learning (Bellemare et al., 2017), which are in principle more sample-efﬁcient than onpolicy learning using PPO since they can reuse samples multiple times.
However, they performed worse than PPO in our experience (Appendix C).
We implement algorithms using the TF-Agents
RL
We employ autoregressive models with one fully-connected layer as policy and value networks since they are faster to train and outperform recurrent networks in our experiments.
At time step t , the network takes as input the W last characters a t − W , ..., a t − 1 that are one-hot encoded, where the context window size W is a hyper-parameter.
To provide the network with information about the current position of the context window, it also receives the time step t , which is embedded using a sinusoidal positional encoding (Vaswani et al., 2017), and concatenated with the one-hot characters.
The policy network outputs a distribution π θ ( a t | s t ) over next the token a t .
The value network V ( s t ) , which approximates the expected future reward for being in state s t , is used as a baseline to reduce the variance of stochastic estimates of equation 1 (Schulman et al., 2017).
[Equation] O PTIMIZATION
Model-based RL learns a model of the environment that is used as a simulator to provide additional pseudo-observations.
While model-free RL has been successful in domains where interaction with the environment is cheap, such as those where the environment is deﬁned by a software program, its high sample complexity may be unrealistic for biological sequence design.
In model-based RL, the MDP M = ( S, A, p, r ) is approximated by a model M (cid:48) =
( S, A, p (cid:48) , r (cid:48) ) with the same state space S and action space
A as M (Sutton & Barto, 2018, Ch.
Since the transition function p is deterministic in our case, only the reward function r ( s t , a t ) needs to be approximated by r (cid:48)
( s t , a t ) .
Since r ( s T , a T ) is non-zero at the last step T and then corresponds to f ( x ) with x =
=
s T − 1 , the problem reduces to approximating f ( x ) .
This can be done by supervised regression: given the data ∪ r (cid:48) <
= r D r from all previous rounds, we ﬁt a regressor f (cid:48) w ( x ) .
Then, the policy is updated ofﬂine using many rounds of optimization on a combination of observations from the simulated environment and actual functional measurements ( x, f ( x )) .
We call our method DyNA-PPO, since it is similar to the DYNA method introduced in Sutton (1991) and employed in Peng et al.
and because it can be used to design DNA.
Model-based RL provides the promise of improved sample efﬁciency when the model is accurate, but it can reduce performance if insufﬁcient data is available for training a trustworthy model.
In this case, the policy is prone to exploit regions where the model is inaccurate (Janner et al., 2019).
To reap the beneﬁt of model-based RL when the model is accurate and avoid reduced performance when it is not, we (i) automatically select the model from a set of candidate models of varying complexity, and (ii) only use the selected model if it is accurate.
After each round of experiments, we ﬁt each of the candidate models on all available data to estimate f ( x ) via supervised regression.
We quantify model accuracy by the R 2 score, which we estimate by ﬁve-fold cross validation.
If the R 2 score of no candidate model is below a pre-speciﬁed threshold τ , we do not perform model-based training in that round.
Otherwise, we build an ensemble model that includes all models with a score greater or equal than τ , and use the average prediction as reward for training the policy. We considered τ as a tunable hyper-parameters, were we found τ = 0 .
5 to be optimal for all problems (see ﬁgure 14. By ignoring the model if it is inaccurate, we aim to prevent the policy from exploiting deﬁciencies of the model (Janner et al., 2019). We perform up to M model-based optimization rounds (see algorithm 1) and stop as soon as the model uncertainty increased by a certain factor relative to the model uncertainty at the beginning of each round. This is motivated by the observation that the model uncertainty is strongly correlated with the model error, and prevents from training the policy with inaccurate model predictions (see ﬁgure 12, 13).
For models, we consider nearest neighbor regression, Bayesian ridge regression, random forests, gradient boosting trees, Gaussian processes, and ensemble of deep neural networks.
Within each model family, we additionally use cross-validation for tuning hyper-parameters, such as the number of trees, tree depth, kernels and kernel parameters, or number of hidden layers and units (see Appendix A.7 for details).
By testing and optimizing the hyper-parameters of different models automatically, the model capacity can dynamically increase as data becomes available.
In Bayesian optimization, non-parametric models such as Gaussian processes are popular regressors, and they also automatically grow model capacity as more data arrives (Shahriari et al., 2015).
On the other hand, with Bayesian optimization there is no opportunity to ignore the regressor entirely if it is unreliable.
Furthermore, Bayesian optimization relies on the ability to do (approximate) Bayesian inference, which in practice is sensitive to the choice of approximation and to hyper-parameter choice (Snoek et al., 2012).
Here, we have ﬁnd it simpler to do cross-validation-based model selection.
Overall, our method combines the positive attributes of both generative and discriminative approaches to sequence design.
Our experiments do not compare to prior work on model-based RL, since these methods primarily focus on estimating a dynamics model for state transitions.
2. D IVERSITY -P ROMOTING R EWARD F UNCTION
Learning policies to generate diverse sequences is important because of several reasons.
In many applications, f ( x ) is an in-vitro (an experiment taking place outside a living organism) surrogate for an in-vivo (reaction occurring inside a living organism) functional measurement that is even more expensive to evaluate than f ( x ) .
The in-vivo measurement may depend on properties that are correlated with f ( x ) and others that are not captured at all in-vitro, such as off-target effects or toxicity.
Therefore, it is desirable for the optimization procedure to discover a diverse set of candidate optima, to improve the chance that a sequence satisfying the ultimate in-vivo criteria is found within this set.
Here, diversity is a downstream metric, for which training the policy π θ ( x ) to maximize equation 1 will not necessarily yield good performance.
For example, a high-quality policy can learn to always generate the same sequence x with a high value of f ( x ) , and will therefore result in zero diversity.
An additional reason that diversity matters is that it yields a good exploration strategy, even for scenarios where optimizing equation 1 is sufﬁcient.
Finally, use of strategies that reward high-diversity policies can reduce the policies’ tendency to generate exact duplicates.
To increase sequence diversity, we employ a simple exploration reward bonus based on the density of proposed sequences, similar to existing exploration techniques based on state visitation frequency (Bellemare et al., 2016).
Speciﬁcally, we deﬁne the ﬁnal reward r T = f
( x ) − λ · dens (cid:15) ( x ) , where dens (cid:15) ( x ) ∈ N + is the weighted number of sequences that have been proposed in previous rounds with a distance of less than (cid:15) away from x , where the weight decays linearly with the distance.
This reward penalizes proposing similar sequences multiple times, where the strength of the penalty is controlled by λ .
As a result, the policy learns not to generate related sequences and hence explores the search space more effectively.
We used the edit-distance as distance metric and tuned the distance radius (cid:15) , where setting (cid:15) > 0 improved exploration on high-dimensional problems (see ﬁgure 11). We also considered an alternative penalty based on the distance of the proposed sequence to already-generated sequences, which we found to be less effective (see ﬁgure 9).
R ELATED WORK
Recently, machine learning approaches have been shown to be effective in optimizing real-world
DNA and protein sequences (Wang et al., 2019; Chhibbar & Joshi, 2019; de Jongh et al., 2019; Liu et al., 2019; Sample et al., 2019; Wu et al., 2019).
Existing methods for biological sequence design fall into three broad categories: evolutionary search, optimization using discriminative models (e.g.
Bayesian optimization), and optimization using generative models (e.g. the cross entropy method).
Evolutionary approaches perform direct local search in the space of sequences.
They include the aforementioned directed evolution and derivatives with application-speciﬁc mutation and recombination steps.
Evolutionary approaches are appealing since they are simple and can easily incorporate human intuition into the design process, but generally suffer from low sample efﬁciency.
Optimization methods based on discriminative models alternate between two steps: (i) using the data that have been collected so far to ﬁt a regressor f (cid:48) w ( x ) to approximate f ( x ) , and (ii) using f (cid:48) w ( x ) to deﬁne an acquisition function that is optimized to select the next batch of sequences.
Recently, such an approach was used to optimize the binding afﬁnity of IgG antibodies (Liu et al., 2019), where a neural network ensemble was used for f (cid:48) w ( x ) .
In general, optimizing the acquisition function is a non-trivial combinatorial optimization problem.
Liu et al.
(2019) employ activation maximization , where gradient-based optimization is performed on a continuous relaxation of the discrete search space.
However, this requires f (cid:48) w ( x ) to be differentiable and optimization of a continuous relaxation is vulnerable to leaving the data manifold (cf.
deep dream (Mordvintsev et al., 2015)).
Bayesian optimization deﬁnes an acquisition function such as expected improvement (Mockus et al., 2014) based on the posterior distribution of f (cid:48) w ( x ) , which enables balancing exploration and exploitation.
Gaussian processes (GPs) are commonly used for Bayesian black box optimization.
See Shahriari et al.
for an overview.
Unfortunately, GPs are hard to scale to large, highdimensional datasets and are sensitive to the choice of hyperparameters.
In response, recent work has performed continuous black box optimization in the latent space of a deep generative model (G ´ omezBombarelli et al., 2018).
However, this approach requires a pre-trained model such as a variational autoencoder to obtain the latent embeddings.
Our model-based reinforcement learning approach is similar to these approaches in that we train a reinforcement learning policy to optimize a model f (cid:48) w ( x ) .
However, our policy is also trained directly on observations of f ( x ) and is able to resort to model-free training by automatically identifying if the model f (cid:48) w ( x ) has insufﬁcient accuracy to be used as a simulation of f ( x ) .
Janner et al.
(2019) investigate conditions in which an estimate of model generalization (their analysis uses validation accuracy) could justify model usage in such model-based policy optimization settings.
Hashimoto et al.
(2018) propose using a cascade of classiﬁers, one per round, to guide sampling progressively better candidates.
Optimization methods based on generative models seek to learn a distribution p θ ( x ) parameterized by θ that maximizes the expected value of f ( x ) : E x ∼ P θ ( x ) [ f ( x )] .
We note that this is the same form as variational optimization objectives, which allow the use of parameter-space evolutionary strategies (Staines & Barber, 2013; Wierstra et al., 2014; Salimans et al., 2017).
Variants of the cross entropy method (De Boer et al., 2005; Brookes et al., 2019a) optimize θ , by alternating two steps: (i) sampling x ∼ p θ ( x ) and evaluating f(x), and (ii) updating θ to maximize this expectation.
Methods differ in how step (ii) is performed.
For example, hillclimb-MLE (Neil et al., 2018) performs maximum-likelihood training on the top k sequences from step (i).
Similarly, Feedback GAN (FBGAN) uses samples whose target function value f ( x ) exceeds ﬁxed threshold for training a generative adversarial network (Gupta & Zou, 2018).
Design by Adaptive Sampling (DbAs) performs weighted MLE of variational autoencoders (Kingma & Welling, 2014), where a sample’s weight corresponds to the probability that f ( x ) is greater than a quantile cutoff under an noise model (Brookes & Listgarten, 2018).
In Brookes et al.
(2019b), p θ ( x ) is further restricted to stay close to a prior distribution over sequences.
An alternative approach for optimizing the above expectation is RL.
While RL has been used for generating natural text (Bahdanau et al., 2016), small molecules (Zhou et al., 2019), and RNA sequences that fold into a particular structure (Runge et al., 2018), we are not aware of applications of
RL to optimizing DNA and protein sequences.
DyNA PPO is related to existing work on model-based RL for sample efﬁcient control (Deisenroth & Rasmussen, 2011; Kurutach et al., 2018; Peng et al., 2018; Kaiser et al., 2019; Janner et al., 2019), with the key difference that state transition function is known and the reward function is unknown in our work, whereas most exist model-based RL approaches seek to model the state-transition function and consider the reward function as known.
Prior work on sequence generation incorporates non-differentiable rewards, like BLEU in machine translation, via weighted maximum likelihood (MLE).
Norouzi et al.
(2016) introduce reward augmented MLE, while Bahdanau et al.
(2016) ﬁne tune an MLE-pretrained model using actor-critic methods.
Reinforcement learning has also been applied to solving combinatorial optimization prob- lems (Bello et al., 2016; Bengio et al., 2018; Dai et al., 2017; Kool et al., 2018).
In this setting sample complexity is less important because evaluating f ( x ) only involves a fast software program.
Recent work has proposed generative models of protein structures (Sabban & Markovsky, 2019) or generative models of amino acids conditional on protein structure (Ingraham et al., 2019). Such methods are outside of the scope of this paper’s experiments, since they could only be used in experimental settings where protein structures, which are expensive to measure, are available. Finally, DNA and protein design differs from small molecule design (Grifﬁths & Hern´andez-Lobato, 2017; Kusner et al., 2017; G´omez-Bombarelli et al., 2018; Jin et al., 2018; Sanchez-Lengeling & Aspuru-Guzik, 2018; Korovina et al., 2019) in the following points: (i) the number of sequences measured in parallel in the lab is typically higher (hundred or thousands vs. dozens) due to the maturity of DNA synthesis and sequencing technology, (ii) the search space is a set of sequences instead of molecular graphs, which require specialized network architectures for both discriminative and generative models, and (iii) molecules must be optimized subject to the constraint that there is a set of reactions to synthesize them, whereas practically all DNA or protein sequences are synthesizable.
E XPERIMENTS
In the next three sections, we compare DyNA-PPO to existing methods on three in-silico optimization problems that we designed in collaboration with life scientists to faithfully simulate the behavior of real wet-lab experiments, which would be cost prohibitive for a comprehensive methodological evaluation.
Along the way, we present ablation experiments to help better understand the behavior of DyNA-PPO.
We compare the performance of model-free policy optimization ( PPO ) and model-based optimization ( DyNA-PPO ) with the following methods that we discussed in Section 3.
Further details for each method can be found in Appendix A
: • RegEvolution : local search based on regularized evolution (Real et al., 2019), which has performed well on other blackbox optimization tasks and can be seen as an instance of directed evolution • DbAs : a cross-entropy optimization approach (Brookes & Listgarten, 2018) with quantile weighting. The generative model is a variational autoencoder with multi-layer perceptron decoder.
• FBGAN : a cross entropy optimization approach using generative adversarial networks (Gupta & Zou, 2018)
• Bayesopt GP :
Bayesian optimization using a GP regressor, expected improvement acquisition function, and activation maximization as acquisition function solver • Bayesopt ENN Bayesopt GP with an ensemble of neural networks regressor • Random : guessing sequences uniformly at random.
We quantify optimization performance by the cumulative maximum of f ( x ) for sequences proposed up to a given round.
We quantify sequence diversity (Section 2.4) in terms of the mean pairwise hamming distance between sequences in a round.
For problems with known optima, we also report the fraction of global optima found.
We replicate experiments with 50 random seeds.
4. O PTIMIZATION OF P ROTEIN C ONTACT I SING M ODELS
We ﬁrst consider synthetic black-box optimization problems based on the 3D structure of naturallyoccurring proteins.
Ising models ﬁt on sets of evolutionary-related protein sequences have been shown to be accurate predictors for proteins’ 3D structure (Shakhnovich & Gutin, 1993;
Weigt et al., 2009; Marks et al., 2011; Sułkowska et al., 2012).
We consider the inverse problem: given a protein, we seek to ﬁnd the amino acid sequence that minimizes the energy of the Ising model parameterized by its structure.
Optimizers are given a budget of 10 rounds with batch size 1000.
The functional form of the energy function is given in Appendix B.1.
On the left of Figure 1 we consider the optimization trajectory for a representative protein and on the right we compare the best f ( x ) found for each method across a range of proteins.
We ﬁnd that DyNA-PPO considerably outperforms the other methods.
We expect that this is because this synthetic reward landscape can be well-described by a model ﬁt using few examples, which also explains the good performance of Bayesian optimization.
On the left of Figure 2 we vary the number of inner-loop optimization rounds of the policy using interaction with the model-based simulated environment, where using 0 rounds corresponds to performing standard PPO.
We ﬁnd that with DyNA-PPO it is worthwhile to spend considerable resources updating the policy using the simulator: the best system performs the most steps.
Doing so is possible because the surrogate model is high quality.
On the right we plot the score of the regressor ﬁt by automated model selection.
Its high accuracy helps DyNA-PPO learn to generate high-quality sequences using very few evaluations of f ( x ) .
4. O PTIMIZATION OF T RANSCRIPTION F ACTOR B INDING S ITES
Transcription factors are protein sequences that bind to DNA sequences and regulate their activity.
In Barrera et al.
(2016), the binding afﬁnity of numerous transcription factors was measured against all possible length-8 DNA sequences ( V = 4).
The resulting dataset deﬁnes 158 different discrete optimization tasks, where the goal of each task is to ﬁnd a DNA sequence of length eight that maximizes the afﬁnity towards one of the transcription factors.
It is well suited for in-silico benchmarking since (i) it is exhaustive and thereby does not require estimating missing f ( x ) and (ii) the distinct local optima of all tasks are known and can be used to quantify exploration (see Appendix B.2 for
[Table]
details).
The optimization methods are given a budget of 10 rounds with a batch size of B = 100 sequences.
We repeat each experiments 50 times with different random seeds.
We use one task (CRX
REF R1) for optimizing the hyper-parameters of all methods, and test performance on 41 hold-out tasks.
In real-world settings, hyper-parameters would either be tuned on in-silico surrogate tasks or would be done using prior in-vitro results from related design tasks.
Figure 3 plots the performance of methods on a single representative binding target (SIX REF R1) as a function of the total number of sequences measured so far.
We ﬁnd that DyNA-PPO and
PPO outperform all other methods in terms of both the cumulative maximum f ( x ) found as well [Figure] as the fraction of local optima discovered.
We also ﬁnd that the diversity of proposed sequences quantiﬁed by the fraction of global optima found is high compared to other generative approaches.
This shows that our method continues to explore the search space by proposing novel sequences instead of converging to a single sequence or a handful of sequences–a desired property as discussed in section 2.4.
Across all tasks DyNA-PPO and PPO rank highest compared with other methods.
We observe the same trend across all 41 hold-out tasks in Table 1.
In Figures 4 and 5 we analyze the effects two key design decisions of DyNA-PPO: model-based training and an exploration bonus.
We ﬁnd that automated model selection automatically increases the complexity of the model, but that the models are not always accurate enough to be used for model-based training.
This explains why DyNA-PPO and PPO are similar in performance.
We also ﬁnd that the exploration bonus outlined in Section 2.4 is more effective than entropy regularization in ﬁnding multiple local optima and promoting sequence diversity.
4. O PTIMIZATION OF A NTI -M ICROBIAL P EPTIDES
Next, we seek to design antimicrobial peptides (AMPs).
AMPs are relatively short (8 - 75 amino acids) protein sequences ( | V | = 20 amino acids) that are promising candidates against multi-resistant pathogens due to their wide range of antimicrobial activities.
We use the dataset proposed by Witten & Witten (2019), which contains 6,760 unique AMP sequences and their antimicrobial activity towards multiple pathogens.
We follow Witten & Witten (2019) for preprocessing the dataset and generating non-AMP sequences as negative training samples.
Unlike the transcription factor binding site dataset, we do not have wet-lab experiments for every sequence in the search space.
Therefore, we ﬁt random forest classiﬁers to predict if a sequence is antimicrobial towards a certain pathogen in the dataset (see Section B.3), and use the predicted probability as the functional measurement f ( x ) to optimize.
Given the high accuracy of the classiﬁer (cross validated AUC > 0.9) we expect that the reward landscape of f ( x ) is of realistic difﬁculty.
We perform 8 rounds with a batch size 250.
Figure 6 compares methods on C. alibicani.
Due to the size of the problem, we found it better to use an ensemble of neural networks as our regressor for BayesOpt.
We ﬁnd that DyNA-PPO does not out-perform PPO, but that they both considerably outperform the other methods in terms of the maximum f ( x ) found.
DyNA-PPO relies on the ability to accurately model the reward landscape using a discriminative model ﬁt on limited examples.
Even though here f ( x ) corresponds to a parametric model ﬁt on data, it may be difﬁcult to approximate, since it was trained on a large about of data using synthetic negative examples that the training within DyNA-PPO does not have access to.
The results demonstrate the useful worst-case behavior of DyNA-PPO.
When the model is unreliable, it gracefully falls back to being standard PPO.
C ONCLUSION
We have shown that RL is an attractive alternative to existing methods for designing DNA and protein sequences.
We have proposed DyNA-PPO, a model-based extension of PPO (Schulman et al., 2017) with automatic model selection that improves sample efﬁciency, and incorporates a reward function that promotes exploration by penalizing identical sequences.
By approximating an expensive wet-lab experiment with a surrogate model, we can perform many rounds of optimization in this simulated environment.
Going forward, we will consider using more sophisticated rewards when optimizing the surrogate to shape the policy to have a variety of desired behaviors.
Finally, while this work has been focused on showing the beneﬁt of DyNA-PPO for biological sequence design, we believe that the large-batch, low-round setting described here may well be of general interest and that model-based RL may be applicable in other domains with long time delays, such as agriculture, education, and economics.
A I MPLEMENTATION D ETAILS A. R EGULARIZED EVOLUTION
Regularized evolution is a variant of directed mutation evolution that regularizes the search by keeping a ﬁxed number of individuals alive as candidates for selection (analogous to death by aging).
At each round, it generates a batch of child sequences by sampling two parent sequences per child from the population via tournament selection, i.e. selecting the ﬁttest out of K randomly sampled individuals.
It then performs crossover of the two parent sequences by copying the characters of one parent from left to right and randomly transitioning to transcribing from the other parent sequence with some crossover probability at each step.
Child sequences are mutated by replacing each token with another from the alphabet with some mutation probability.
For variable length sequences, we also allowed insertion and deletion mutations.
As hyper-parameters, we tune the tournament size, substitution-, insertion-, and deletion-probabilities.
A. MCMC AND S IMULATED A NNEALING
MCMC and simulated annealing (Kirkpatrick et al., 1983) resemble evolution with no crossover, and selection only occurring between an individual and its parent. Beginning with a random population, each individual evolves as a single chain, with neighborhood structure deﬁned by the mutation operator described in section A.1. The chain for a given individual x , always accepts the transition to a child x (cid:48) if the rewards f ( x (cid:48) ) >
f ( x ) .
If the child has a lower reward than the parent, then we accept the transition with some acceptance probability.
For MCMC, the acceptance probability is f ( x (cid:48) ) /f
( x ) , while for simulated annealing it is exp(( f ( x (cid:48) )
− f ( x )) /T )
for some temperature T.
Low temperatures decrease the likelihood of accepting a move which perform worse than the parent.
When a transition is rejected, the next mutation on the chain begins from x , while if accepted it begins from x (cid:48) .
One may additionally anneal the temperature over time to decrease exploration in favor of exploitation.
A. F EEDBACK GAN
We follow the methodology suggested by Gupta & Zou (2018).
Instead of using a constant threshold for selecting sequences for training as described in the original publication, we used a quantile cutoff, which does not depend on the absolute scale of f ( x ) and performed better in our experiments.
As hyper-parameters, we tuned the quantile cutoff, learning rate, batch size, discriminator and generator training epochs, the gradient penalty weight, the gumble softmax temperature, and the number of latent variables of the generator.
A. D B A S
We follow the methodology suggested by Brookes & Listgarten (2018).
As hyper-parameters, we optimized the quantile for selecting training samples, learning rate, batch size, training epochs, number of hidden units of the MLP generator and discriminator, and number of latent variables.
The generative model is an variational autoencoder with a multi-layer perceptron decoder.
We also considered DbAs with a LSTM as generative model, which performed worse on all problem (see ﬁgure 8).
A. B AYESIAN O PTIMIZATION
As regressors, we considered a Gaussian process (GP) with RBF kernel on one-hot features, and an ensemble of ten fully-connected neural networks with one fully connected layer and 128 hidden units.
We used the regressor output to compute the expected improvement or posterior mean acquisition function, which we maximized by gradient ascent for a certain number of acquisition steps using activation maximization.
We took the resulting B unique sequences with highest acquisition function value as sequences to measure in the next round.
We tuned the length scale and variance of the RBF kernel, and the learning rate, batch size, and number of training epochs of the neural network ensemble.
We further tuned the number of gradient ascent steps for activation maximization.
A. PPO AND D Y NA PPO
We used the PPOAgent from the TF-Agents RL library (Guadarrama et al., 2018) for policy training.
After each round, we trained trained the agent on the collected batch of sequences for a relatively high number of steps (about 72), since it resulted in performance increase.
We used the PPO-Penalty instead of PPO-Clip, since it performed slightly better in our experiments Schulman et al.
We used a policy and value network with one fully connected layer and 128 hidden units.
We set the context window W of the autoregressive model to the minimum of the total sequence length and 50.
As hyper-parameters, we tuned the learning rate, number of training steps, adaptive KL target, and entropy regularization.
For DyNA PPO, we also tuned the minimum score τ (see section 2.3) for model-based optimization.
A. A UTOMATED M ODEL S ELECTION
For automatic model selection, we optimized the hyper-parameters of model candidates by ﬁvefold cross-validation using the scikit class RandomizedSearchCV, and quantiﬁed model accuracy by the R 2 score.
To account for randomness in the R 2 score between models due to different crossvalidation splits, we used the same split for evaluating each of the models per round.
We considered the following candidate models (implemented in scikit learn) and corresponding hyper-parameters:
• KNeighborsRegressor: n neighbors • BayesianRidge: alpha 1, alpha 2, lambda 1, lamdba• RandomForestRegressor: max depth, max features, n estimators • ExtraTreesRegressor: max depth, max features, n estimators • GradientBoostingRegressor: learning rate, max depth, n estimators • GaussianProcessRegressor: with RBF, RationalQuadratic, and Matern kernel
We also considered an ensemble of 10 neural networks with two convolutional layers and one fully connected layer, and optimized the learning rate and number of training epochs.
B D ATASET D ETAILS
B
Given a protein in the Protein Data Bank (Berman et al., 2003), we compute the energy E ( x ) for sequence x as E ( x ) = (cid:80) i φ i ( x i ) + (cid:80) ij C ij φ ( x i , x j ) , where x i refers to the value in the i -th position of sequence x .
C ij is an indicator for whether the Cα atoms of the residues at positions i and j are separated by less than 8 Angstroms when the protein folds. φ ( x i , x j ) is a widely-used ‘pair potential’ based on co-occurence probabilities derived from the structures of real-world proteins (Miyazawa & Jernigan, 1996).
The same 20 x 20 table of pair potentials is used at all positions in the sequence, and thus the difference in energy functions across proteins is dictated only by their differing contact map structure.
We set the local term φ i ( x i ) to zero.
In future work, it would be interesting to consider non-zero local terms.
Our experiments consider a set of qualitatively-different proteins listed at the bottom-right of Figure 1.
We identify the local optima using the same procedure as in Section B.2, except without accounting for reverse complements.
B. T RANSCRIPTION FACTOR BINDING SITE DATASET
We used the dataset described by Barrera et al.
(2016), and normalized binding afﬁnities between zero and one.
To reduce computational costs, we only considered the ﬁrst replicate (REF R1) of each wild type transcription factor in the dataset, which resulted in 41 optimization targets that we used for comparing optimizers as described in Section 4.2.
We extracted local optima for each binding target as follows.
First, we separated sequences into forward and reverse sequences by ordering sequences lexicographically and including each sequence in the set of forward sequences unless the set already contained its reverse complement.
We then chose the 100 forward sequences with the highest binding afﬁnity and clustered them using the hamming distance metric, where we determined the number of clusters by ﬁnding the number of PCA components required to explain 95% of variance.
We then used the sequences with the highest reward per cluster and their reverse complement as local optima.
Our goal was to construct a metric for this small exhaustive dataset with which to judge the ability of the algorithms to ﬁnd distinct sequences with high scores.
We do not use a typical motif-ﬁnding approach popular in the literature for identifying transcription factor binding sites, since we seek to ensure generalizability of these algorithms to a wide range of use cases.
B. A NITMICROBIAL P EPTIDE D ATASET
We downloaded the dataset 1 provided by Witten & Witten (2019), and followed the paper for preprocessing sequences and generating non-AMP sequences as negative training samples.
We additionally excluded sequences containing cysteine and sequences shorter than 15 or longer than 50 amino acids.
We ﬁt one classiﬁer to predict if a sequence is antimicrobial towards either E.coli, S.aureus,
P.aeruginosa, or B.subtilis, which we used for hyper-parameter tuning, and a second classiﬁer for C. alibicani, which we used for hold-out evaluation.
We used C. alibicani as hold-out target since its antimicrobial activity was least correlated with the activity of other pathogenes in the dataset with more than 1000 AMP sequences.
We used random forest classiﬁers since they were more accurate (cross validated AUC 0.99 and 0.94) than alternative models such as k-nearest neighbors, Gaussian processes, or neural networks.
Since sequences are variable-length, we padded them to the maximum sequence length of 50 and extended the vocabulary by an additional end of sequence token.
Tokens after the ﬁst end of sequence token were ignored when evaluating f ( x ) . C C OMPARISION OF RL METHODS [Figure]
DyNA-PPO is built on PPO, which we have found to outperform other policy-based and value-based
RL methods in practice on our problems.
In Figure 7 we contrast the performance of PPO (Schulman et al., 2017), REINFORCE (Williams, 1992), deep Q-learning (Mnih et al., 2015), and categorical distributional deep Q-learning (Bellemare et al., 2017) on all problems considered in section 4.
We ﬁnd that PPO has better exploration properties than REINFORCE, which tends to converge too soon to a local optimum.
The poor performance of DQN and CatDQN can be explained by the sparse reward (the reward is only non-zero at the terminal state), such that the Bellman error and training loss for updating the Q network are zero in most states.
We also found the performance of DQN and CatDQN to be sensitive to the choice of the epsilon greedy rate and Boltzmann temperature for trading-off exploration and exploitation and increasing diversity.
D C OMPARISON OF ADDITIONAL BASELINES
[Figure]
[Figure]
F A NALYSIS OF MODEL - BASED TRAINING
[Figure]
[Figure]