A reproducibility study of “Augmenting Genetic Algorithms with Deep Neural Networks for Exploring the Chemical Space”
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
Nigam et al.
(2020) report a genetic algorithm (GA) utilizing the SELFIES representation (Krenn et al., 2020) and alsopropose an adaptive, neural network-based penalty that is supposed to improve the diversity of the generated molecules.
The main claims of the paper are that this GA outperforms other generative techniques (as measured by the penalizedlogP) and that a neural network-based adaptive penalty increases the diversity of the generated molecules.
Methodology
We re-used the code published by the authors after minor refactoring and re-ran the key experiments on a typicalworkstation (two 16 core Intel Xeon Gold 5218s, Quadro RTX 6000) within two weeks using more recent versions ofthe dependencies.
In particular, we used a new, major version of the SELFIES library and also quantiﬁed the diversityof the generated molecules and the effect of different hyperparameters.
All of our experiments were tracked on the
Weights and Biases platform (Biewald, 2020).
Results
Overall, we were able to reproduce comparable results using the SELFIES-based GA—but mostly by exploitingdeﬁciencies of the (easily optimizable) penalized logP ﬁtness function (i.e., generating long, sulfur-containing chains).
In addition, we also reproduce results showing that the discriminator can be used to bias the generation of molecules toones that are similar to the reference set.
Moreover, we propose a new similarity-based adaptive penalty that outperforms the original algorithm on the penalizedlogP score.
We perform ablation studies and show that the complexity of a multilayer network discriminator, as used inthe original work, is not crucial to reproduce the dependence of the penalized logP on the penalty weight.
Importantly,we emphasize the need for a more representative comparison between algorithms. We analyze the performance of theoriginal algorithm on the Guacamol benchmark set and ﬁnd that it tends to show low intra-generation diversity.
What was easy
Reproducing all the key results (including the plots) was easy since the authors provided code with pre-deﬁned settings and useful comments for every relevant experiment.
Hence, it did not require complete implementation from scratch
What was difﬁcult
Without the provided code reproducing some parts of the papers would have been signiﬁcantly more time-consuming
Communication with original authors
Introduction
The accelerated discovery of new materials and molecules requires efﬁcient techniques to explore chemical space.
Sincechemical design space is vast, simple enumeration and brute-force screening approaches with experimental testing areunfeasible.
For example, Polishchuk et al.
(2013) estimated the number of drug-like molecules to be between 10 23 and10 60 .
To address this problem, generative techniques such as generative adversarial neural networks (GANs) (Goodfellowet al., 2014; De Cao and Kipf, 2018), variational autoencoders (VAEs) (Kingma and Welling, 2013; Gómez-Bombarelliet al., 2018; Kusner et al., 2017; Liu et al., 2018) and genetic algorithms (GAs) (Devillers, 1996; Supady et al., 2015)have received mounting attention from the chemistry community as means to efﬁciently perform “inverse design”(Sanchez-Lengeling and Aspuru-Guzik, 2018).
That is, rather than exhaustively searching through chemical space,inverse design allows one to optimize chemical design based on a desired target property.
A limitation of theseapproaches, however, lies in the fact that they often use a conventional string-based molecular representations known assimpliﬁed molecular-input line-entry notation system (SMILES), which require careful treatment to ensure the validityof the generated molecules, as most SMILES do not correspond to valid molecules.
To remedy this problem, Krennet al.
(2020) proposed SELFIES, a string representation for which the random arrangement of characters correspondsto a valid molecules.
This property has been recently exploited by Nigam et al.
(2021) to efﬁciently interpolate inchemical space and was also proposed by Nigam et al.
(2020) for use in GAs.
Similar to the previous works of Jensen (2019) and Brown et al.
(2019), the results by Nigam et al.
(2020) suggestthat conventional approaches such as GA can outperform generative machine learning models and are therefore animportant reference for future advancement in the ﬁeld.
Scope of reproducibility
We chose to address the following main claims from the original paper:
Claim 1: Genetic algorithms operating on SELFIES outperform state-of-the-art generative models Nigam etal.
showed that their algorithm could achieve higher penalized logP scores than all references they considered.
Thepenalized logP score is hereby deﬁned as logP coefﬁcient penalized with the synthetic accessibility score (SA), and aring penalty
[Equation]
The logP coefﬁcient is an estimator of the solubility, synthetic accessibility score is a heuristic reported by Ertl and
Schuffenhauer (2009) that estimates the ease of synthesis (a lower score means easier synthetic accessibility), and thering penalty penalizes cycles larger than six.
Hence, molecules with higher J ( m ) are thought to have a more drugableproﬁle than those with low J ( m ) .
In addition, Nigam et al.
also demonstrated that a GA on SELFIES can perform wellon optimization of other objectives such as similarity-constrained optimization or optimization of two conﬂicting goals.
Claim 2: A neural network derived discriminator term, D ( m ) , added to ﬁtness ( J ( m ) + β D ( m ) ) increases thediversity of the generated molecules Nigam et al.
did not quantify the diversity of the generated molecules beyondan unsupervised analysis of the GA trajectory, but achieved higher penalized logP scores than compared to thosewithout the discriminator term—especially when employed in a time-adaptive setting.
In addition to analyzing the reproducibility of these claims, we also attempted to quantify the evolution of the diversity,understand the inﬂuence of some hyperparameters, and propose improvements to the adaptive penalty.
Methodology
We based our replication study on the code provided by Nigam et al.
We used one workstation to run all experiments,which are mostly CPU-bound (requiring around 3 s to 8 s per GA generation on our workstation).
The code we used,including the analysis code, is available in a GitHub repository.
We re-used the GA implementation from the original paper and the discriminator model architecture (two-layer feedforward network with sigmoid activations trained on the binary cross entropy loss using the Adam optimizer (Kingma and Ba, 2015) with learning rate set to 0 .
001 and weight decay set to 1 × 10 − 4 ; model artifacts are available on the
Weights and Biases platform).
Nigam et al.
(2020) did not report how they selected this architecture and parametersetting.
For this reason we ran additional experiments using logistic regression as the discriminator model.
3.2 Datasets
We used the ZINC dataset (Irwin et al., 2012) provided in the GitHub repository associated with the original paperfor some baselines and constrained improvement experiments.
4 The J ( m ) scores are normalized with respect to thisdataset (logP: mean 2.47, standard deviation 1.42, SAS: mean 3.05, standard deviation 0.831, ring penalty: mean 0.038,standard deviation 0.224).
3.3 Hyperparameters
Details about hyperparameter tuning were not provided in the original paper.
Hyperparameters were set for all experiments in the code provided by the original reference and were re-used in this work.
Furthermore, we performedsome additional experiments in this work to better understand the inﬂuence of the β parameter, model architecture, andtraining setup.
3.4 Experimental setup
The code used for this study is available on GitHub 5 .
All experiments (including the failed attempts) were trackedusing the Weights and Biases platform 6 .
3.5 Computational requirements
The experiments do not necessarily need to be run on GPU for high performance, since only some of them use theneural network-based discriminator and since the model that is employed is small.
The computational burden lies muchmore in the CPU-bound GA operations.
Each GA generation takes 3 s to 8 s to run on one core (note that the originalimplementation already included some parallelization for the score calculation, and we used a population size of 500for all experiments).
Also the process memory requirements are low ( < 100 MB ).
It is practical to parallelize thescore evaluations, especially for the Guacamol benchmark.
Exact timings for all experiments and traces of hardwareutilization can be found in the dashboard on the Weights and Biases platform.
Results and Discussion
Based on the code provided by the authors, we could reproduce results showing that a SELFIES-based GA can createmolecules with high J ( m ) (numerically higher than the references considered by Nigam et al.
and that theadaptive penalty can bias the generation of molecules that are similar to a reference set.
We could also show that the
SELFIES-based GA can outperform relevant baselines in some aspects but lacks intra-population diversity.
4.1 Baselines
Nigam et al.
computed the penalized logP score for random SELFIES and found an impressively high score of6 .
19 ± 0 .
63 , outperforming many other generative techniques such as VAEs, (Gómez-Bombarelli et al., 2018; Daiet al., 2018) and Monte-Carlo Tree Search (MCTS)
(Yang et al., 2017).
We were able to reproduce the result whenusing the exact same version of the SELFIES library (v0.1.1).
However, we obtained considerably lower scores whenrunning this experiment using the latest version of the SELFIES library (v1.0.2), for which some bugs have been ﬁxedand modiﬁcations such as kekulization as well as an extended alphabet have been added (i.e., more complex moleculescan be assembled).
To ensure compatibility of the latest SELFIES version with the old one, we ran experiments usingthe same alphabet (of 21 characters) as in the original study and an extended alphabet (of all 61 semantically robustcharacters), while keeping the constraints in both experiments to a maximum of 81 characters.
our
2, original alphabet)
[Figure]
The reasons for the high baseline score reported in the original paper is clear when one compares a few randomlysampled molecules: Using the original approach, the molecules have a higher density of aromatic rings, which increasesthe penalized logP score.
We suspect this high baseline is achieved due to an inductive bias introduced by the code, as we have found that some
SELFIES created with the 21 character alphabet could not be decoded into valid SMILES using SELFIES version 0.1.1.
We could decode those SELFIES using v1.0.2.
and found J ( m )
= − 4 .
30 ± 2 .
73 (averaged over 50 000 SELFIES thatraised errors in v0.1.1.
but could be successfully decoded into SMILES without RDKit warning/error with v.1.0.2.)
For this reason, we used version 1.0.2 of the code for all subsequent experiments.
More importantly, our results showthat choosing a relevant baseline is nontrivial as, for example, the choice of the alphabet can be thought of as inductivebias.
Hence, following Brown et al.
(2019), we suggest that the “best of dataset” is a more relevant baseline as anygood generative model should outperform the training set or starting population.
Furthermore, we ﬁnd that comparingalgorithm performance based on penalized logP scores without setting a limit to molecular size can be misleading, asthis score can be maximized by simply increasing the molecular size, whereas from a practical point of view such largemolecules have limited potential as a drug molecule (for example, very large logP values can lead to pharmakokineticproblems).
We note that Nigam et al.
(2020) did apply a relatively high threshold limit to the maximum molecular size(81 characters).
In contrast, Jensen (2019) applied a threshold of 39 .
15 ± 3 .
50 non hydrogen atoms for their GA.
4.2 SELFIES GA without discriminator (Claim 1)
Using the GA, Nigam et al.
(2020) could achieve nearly double the highest reported penalized logP score reported sofar by evolving SELFIES seeded with a methane molecule.
As noted earlier, Nigam et al.
applied a higher thresholdon the maximum number of heavy atoms than some other previous works.
We ﬁnd a comparable score of J ( m ) =11 .
911 ± 1 . 262 (averaged over 10 runs) using the same approach as in the original paper, albeit the highest-scoringmolecules have a heavy atom count of 58 .
800 ± 9 .
[Figure]
4.3 SELFIES GA with discriminator (Claim 2)
In the original paper, the discriminator neural network is trained jointly on mutated SELFIES strings, and data from thereference as a binary classiﬁer, where molecules from the reference set are labeled with 1.
Increasing β hence meansbiasing the GA to create molecules that are similar to the reference set.
This also means that molecules that survivefor longer times will tend to reduce D ( m ) .
The intuition behind this is that a large discriminator term will force thechemistry to be similar to the reference set and cause long-surviving families to die.
Note that in this study we alsoinvestigated negative β , which effectively penalizes molecules that are similar to the reference database.
Similar to Nigam et al.
we ﬁnd that large β yields J ( m ) around zero (Figure 3), as the scores are normalized withrespect to the reference set.
From Figure 4, we can clearly see that if we reward similarity to the reference set (large β ),the generated molecules are more complex and contain a larger variety of functionalities.
It also interesting to use large,negative β .
This effectively penalizes similarity to the reference set.
For β = − 100 we observe that the GA generatessmall molecules, e.g. nitrous oxide, that often do not contain any carbon.
Our GA runs outperform the ones reported in the original paper in terms of J ( m ) , but also in our case, the GA exploitsdeﬁciencies in the scoring function.
4.3.1 Internal diversity quantiﬁcation
Nigam et al.
emphasize the importance of diversity in the generative molecules in qualitative terms.
It is instructive toquantify how the different generative approaches discussed in this work inﬂuence the diversity.
To do this
, we followthe approach proposed by Benhenda (2017) and calculate the mean pairwise Tanimoto distance T d ( x, y ) of the Morganﬁngerprints (Rogers and Hahn, 2010) of the set
A best-performing molecules obtained in the last generation, that is
[Equation]
From Figure 5 we see that a low β increases the similarity between subsequent best performing molecules within ageneration, whereas high, positive β decreases the similarity between the highest-scoring molecules—but also forces
J ( m ) to be near the mean of the reference set (Figure 3).
For some applications it is important to generate a diverse set of high performing molecules, for example, as in the caseof drug discovery where the chances of commercialization of an early lead structure are quite low.
For this reason,
we
[Figure] [Figure]
generation 0 .0 .0 .
.m
e a n T a n i m o t
o s i m il a r i t
y l a s t ﬁ v e g e n e r a t i o n s a)   generation
nu m b e r o f h e a vy a t o m s b) β - -50- -0.
analyzed how many unique and diverse molecules the GA generates.
From Figure 6 we see that the fraction of uniquemolecules and the diversity within a generation is decreasing as the optimization progresses.
For the fraction of uniquemolecules, we ﬁnd that all positive and small negative values of beta β show similar behaviors, yielding about 40 %unique molecules after 500 generations.
Only the lowest values of β do not reach more than 20 % unique molecules.
In terms of intra-population diversity, we ﬁnd that high values of β biases the generation to more diverse populations.
4.4 Time-adaptive penalty (Claim 2)
A typical issue with GAs is that they can get stuck and only return the same molecule for multiple iterations.
Tomitigate this problem, Nigam et al.
proposed a time-adaptive penalty in which they added a 1000 D ( m ) term to theﬁtness function only if the maximum ﬁtness stagnated at the exact same maximum ﬁtness value for the last ﬁvegenerations.
The authors could signiﬁcantly increase the maximum J ( m ) found by the GA using this approach but didnot justify their choice of β .
We also performed this experiment with different β and quantiﬁed the evolution of internaldiversity and molecular sizes.
From Figure 7 we ﬁnd that the time-adaptive penalty frequently reduces the J ( m ) andmean Tanimoto similarity, after which both quickly recover.
This, on average, reduces the mean Tanimoto similarity ofthe best performing molecules with increasing β .
Notably, this average is higher than for the case in which the sameβ is applied all the time.
By analyzing multiple independent runs we can quantify how the generator stagnates underdifferent β .
Stagnation is deﬁned here in the same way Nigam et al.
deﬁnes it in their code, i.e., if maximum J ( m ) hasnot changed for ﬁve generations.
From Figure 8, we see that runs at low β have a higher chance of getting stuck for asubstantial number of generations due to the penalty not having a large enough effect on the ﬁtness function.
In onecase ( β = 200), the generator stagnated for 80 % of generations.
4.5 Constrained optimization
A common benchmark task for generative models is to generate molecules that are similar to some target.
Nigam etal.
adapted the experimental framework from You et al.
(2018) in which one aims to improve the J ( m ) for 800 lowperforming structures from the references set.
Similar to Nigam et al.
we performed this experiment with two similarity
generation 0 .0 .0 .
.f
r a c t i o n o f un
i q u e m o l ec u l e s a)
[Figure]
generation m e a n T a n i m o t
o s i m il a r i t y b) thresholds δ = { 0 . 4 , 0 . 6 } between the Morgan ﬁngerprints of the target molecule and the GA output, but limited ourstudy to 285 molecules for δ = 0 .
4 and 120 for δ = 0 .
Overall, our ﬁndings agree with the ones of Nigam et al.
, eventhough we observe slightly lower success rates (we count a run as success when the improvement in the score > 0 ).
More speciﬁcally, we fail to improve J ( m ) for more molecules than Nigam et al.
and could not achieve the perfectsuccess rate of 100 % .
For a similarity threshold of δ = 0 .
4 , we ﬁnd an average improvement of 4 .
11 ± 1 .
58 and asuccess rate of 99 % .
For the larger tolerance of δ = 0 .
6 , we ﬁnd an average improvement of 1 .
87 ± 1 .
32 and a successrate of 100 % .
We ﬁnd that the mean improvements agree to within the error margins of those reported by Nigam et al.
4.6 Optimization of multiple properties
To demonstrate the versatility of their approach, Nigam et al.
also incorporated the quantitative estimate of drug-likeness(QED) score (Bickerton et al., 2012) into their loss function.
This is an interesting problem as solubility indicator, logP,and the QED score cannot be maximized at the same time, i.e., one would like to recover the Pareto front.
From
Figure 9a)
we see that the best performing molecules in a generation quickly converge to one point of the Paretofront, i.e., after 100 generations we do not observe much evolution of the properties of the top scoring molecules.
From
Figure 9b)
we ﬁnd that the molecules in the ﬁnal generation have a good Pareto front coverage, but only in few instancesoutperform the original Pareto optimal molecules from the ZINC dataset (we ﬁnd the hypervolume of the Pareto frontof the ZINC database to be 7.37 and the one found by the GA to be 7.14 w.r.t.
the nadir point of the ZINC dataset).
4.7 Similarity-triggered adaptive penalty
Nigam et al.
used the time-adaptive penalty to encourage explorative behavior.
They measure estimated stagnationbased on constant maximum ﬁtness.
We anticipated that better results with greater molecular diversity could be achievedby using a penalty that is applied based on an explicit structure similarity criterion.
Therefore, we implemented anadaptive penalty based on the internal molecular similarity of the best-performing molecules in the last ﬁve generations(measured in terms of the average pairwise distances of the Morgan ﬁngerprints (Rogers and Hahn, 2010) as proposed
[Figure]
J ( m ) a)
[Figure]
generation 0 .0 .0 .
.m
e a n s i m il a r i t
y l a s t ﬁ v e g e n e r a t i o n s b) β
generation
nu m b e r o f h e a vy a t o m s c)
[Figure]
n e r a t i o n s s t ag n a t e d
[Figure]
by Benhenda (2017)).
In contrast to Nigam et al., who started to impose this penalty after 100 iterations, we startedimposing this adaptive penalty after 20 generations.
We did this based on the consideration that after 20 generations,the GA should have enough time to evolve away from the methane starting point to some decently performing molecule(see for example Figure 9a); note that we did not tune this hyperparameter).
From Figure 10 we see that the success with this penalty highly depends on the similarity threshold.
If it is chosen toosmall (i.e., close to non-signiﬁcant following Vogt and Bajorath (2020))
the ﬁtness does not improve.
We ﬁnd that byusing the similarity-triggered penalty we can keep the similarity between subsequent top-performing molecules under auser-deﬁned threshold.
Interestingly, we also observe that this does not necessarily compromise J ( m ) .
We ﬁnd scorescomparable to the highest ones shown in Figure 3—however, we do not converge to a similarity of close to unity usingthe similarity-triggered penalty.
4.8 Inﬂuence of the labeling convention
In the setup proposed by Nigam et al.
long-surviving molecules will return low discriminator scores D ( m ) .
Oneextension of this approach can be to penalize long-surviving molecules, i.e., subtract a penalty from the ﬁtness formolecule classes that are long-surviving.
This can be achieved by reversing the way in which the discriminator modelis trained, i.e., predicting a score > 0 .
5 for long-surviving molecules (instead of 0 in the original approach).
Figure 11shows some results under this framework.
We see that with the reversed labeling convention, J ( m ) is not as sensitiveto changes to β than compared with the labeling convention chosen by Nigam et al.
This is also consistent withwhat we observe in the evolution of the discriminator scores (Figure 12).
The reversed labels fail to reach this objective(for large β in the original labeling convention we can bias the GA to the generation of molecules with large D ( m ) ) .
We suspect that this behavior occurs, because it is easier for the model to learn the similarity to the reference set thanit is to learn the stagnation, potentially due to low intra-population diversity.
These observations might imply that thediscriminator term rather works via biasing to similarity to a reference set rather than by penalizing long-survivingmolecules.
To understand this behavior better, we compared the evolution of the best performing molecules for different β .
We ﬁndthat the D ( m ) penalty encourages exploratory behavior in the ﬁrst few generations.
After 100–200 generations, the GAbegins to recognize patterns (long chains with sulfur) that it can easily exploit and outperforms J ( m ) that have been
[Figure]
s i m il a r i t y b)
[Figure]
h e a vy m e t a l c o un t c)
[Figure]
i t y
[Figure]
[Figure]
h e a vy m e t a l c o un t c)
[Figure]
[Figure]
generation c) previously reported.
Again we note, however, that this is mainly attributed to the larger molecular size (see Figure 11c).
For large β (i.e., encouraging similarity) we ﬁnd that the scaffold mainly evolves by elongating linear chains.
In comparison to Figure 4, we also observe that molecules do not contain functionalities as complex as those found forlarge β with the original labeling, with which we can explicitly reward similarity to the reference set.
Note that withthe ﬂipped labels, the molecules from the reference set will yield a discriminator score of D ( m ) = 0 .
4.9 Inﬂuence of discriminator model architecture
Nigam et al.
did not rationalize the choice of the model architecture.
To understand the sensitivity of the algorithmperformance to the architecture of the discriminator model, we replaced the neural network with a logistic regressionmodel (i.e., we use only one hidden layer) and ran experiments with different β .
From Figure 14, we see that the simplermodel can also be used successfully as a discriminator and reproduce the same dependence of J ( m ) on β as seen inthe runs with neural network-based discriminator model.
Analysis of the evolution reveals an interesting differencecompared to the deeper model—the logistic regression performs worse in penalizing long-surviving molecules.
In otherwords, we observe higher similarities of consecutive molecules for high β .
4.10 Performance on the GuacaMol benchmark
As a subsequent benchmark study, we tested the GA on the goal-directed benchmarks from the GuacaMol library,which have been shown to be more challenging than the optimization of simple molecular properties such as J ( m )(Brown et al., 2019).
Furthermore, to test whether the SMILES-based GA (Yoshikawa et al., 2018) can improve overthe “best of dataset” within the GuacaMol benchmark suite, we initialized the GA with the best scoring moleculesfrom the GuacaMol dataset.
Note that we did not perform an exhaustive and systematic hyperparameter optimization 7 ,wherefore the scores (Table 1) represent a lower limit on the performance of the SELFIES-GA.
[Figure]
generation
[Figure]
[Figure] [Figure]
) β
[Figure] [Figure]
.1 .s
i m il a r i t
y
[Figure] [Figure]
generation −
[Figure] [Figure]
[Figure]
We did not ﬁnd the SELFIE-bases GA to outperform the best performances in the leaderboard, but—even withoutsystematic parameter tuning—we did ﬁnd it to mostly improve beyond the “best of dataset” benchmark, and in somecases (e.g., Celecoxib rediscovery), also to perform better than the grammatical evolution of SMILES reported by
Yoshikawa et al.
(SMILES-GA baseline).
Most striking are the lower scores on the isomer discovery tasks(C 11 H 24 , C 9 H 10 N 2 O 2 PF 2 Cl).
Even with discriminator-based penalty the GA returns only few unique molecules, whichreﬂects the ﬁndings shown in Figure 6 in which the GA converges to a low number of unique molecules.
In the caseof isomer discovery, this issue is exacerbated since all valid isomers will have the same ﬁtness, and hence the sameprobability of being replaced with mutations on one of the top-scoring molecules.
Additionally, there is no term thatencourages diversity within a population.
For these reasons, we observe that the isomer discovery scores improve if werun the GA for fewer iterations (we see from Figure 6 that the fraction of unique molecules decreases over the course ofthe optimization).
On the other hand, if only top-1 scores are considered (as in the rediscovery tasks) the SELFIES-GAoutperforms the SMILES-GA baseline.
For these tasks, we see that running the GA for more iterations improves thescores.
Conclusions and Future Work
In summary, we could easily reproduce high performance in the optimization of the penalized logP score using the
SELFIES based GA.
We could also reproduce behavior showing that an adaptive penalty can be used to bias the GA tothe generation of molecules that are similar to a reference database.
Building on top of the work of Nigam et al., wealso quantiﬁed the evolution of similarity for different β and investigated the inﬂuence of the discriminator model, thelabeling convention, and proposed a new way of applying the time-adaptive penalty, which we found to outperform theoriginal results.
However, our analysis also highlights the need for fairer comparisons when benchmarking generative algorithms.
The
GA mostly outperforms the scores reported so far mainly by exploiting deﬁciencies of the J ( m ) scoring function,which results in the generation of long molecules with unfeasible substructures (e.g., sulfur chains).
For this reason, wealso benchmarked the performance of the algorithm on the GuacaMol set of benchmarks.
Clearly, such benchmarksallow for a much more holistic and fairer comparison of algorithms than the use of the penalized logP score.
The issuewith the latter is not only that it is easy to exploit, but that authors use different thresholds on the maximum number ofatoms or different character sets which can make it more challenging to reproduce results.
Notably, we observed that there is often a relatively low diversity in the molecules within a generation—which leadsto low scores on some of the GuacaMol benchmarks (in which not only the top-1 score is considered).
Since a largepool of diverse molecules is quite relevant for applications with high failure rates (e.g., drug discovery), we propose theaddition of penalty terms of the form 1 − internal population diversity to the ﬁtness so that the GA is biased towarda higher intra-generation diversity.
Acknowledgments
Our study made use of the following libraries: ccbmlib (Vogt and Bajorath, 2020), click (Ronacher, 2021), GuacaMol(Brown et al., 2019), jupyter (Kluyver et al., 2016), matplotlib (Hunter, 2007), numpy (Harris et al., 2020), pandas(McKinney, 2010), pyepal (Jablonka et al., 2020), pytorch (Paszke et al., 2019), RDKit (Landrum et al.), SciencePlots(Garrett, 2020), scikit-learn (Pedregosa et al., 2011), scipy (SciPy 1.0 Contributors et al., 2020), SELFIES (Krenn et al.,2020).