A Fresh Look at De Novo Molecular Design Benchmarks
Abstract
De novo molecular design is a thriving research area in machine learning (ML) that lacks ubiquitous, high-quality, standardized benchmark tasks.
Many existing benchmark tasks do not precisely specify a training dataset or an evaluation budget, which is problematic as they can signiﬁcantly affect the performance of ML algorithms.
This work elucidates the effect of dataset sizes and experimental budgets on established molecular optimization methods through a comprehensive evaluation with 11 selected benchmark tasks.
We observe that the dataset size and budget signiﬁcantly impact all methods’ performance and relative ranking, suggesting that a meaningful comparison requires more than a single benchmark setup.
Our results also highlight the relative difﬁculty of benchmarks, implying that logP and QED are poor objectives.
We end by offering guidance to researchers on their choice of experiments.
Introduction
In recent years, de novo molecular design has seen increased attention from the ML community.
The goal is to produce data-driven algorithms that can effectively and efﬁciently guide the design of novel molecules and materials with desirable properties such as drug efﬁcacy.
Mathematically the problem is typically posed as the optimization of a black-box objective function over molecular space.
For real-world problems, evaluating these objective functions typically involves costly and time-consuming computational or wet-lab experiments; for this reason, researchers often assess their algorithms on fast and approximate computational objectives instead.
The community still lacks ubiquitous benchmarks (like Imagenet [Deng et al., 2009] for instance) and many nascent benchmarks suffer from severe limitations, such as
Unrealistic Optimization Objectives : Simply maximizing molecular properties such as the logP, a metric for the solubility of a compound in non-polar solvents, or the “Quantitative Estimate of Druglikeness” (QED) have limited medicinal value.
Further, such objectives are more manageable than real-world de novo molecular design tasks and can be easily solved with basic random-search-like algorithms (see Brown et al.
[2019] and Section 3).
Lack of Canonical Datasets : The size and composition of the training set are known to impact the performance of virtually every ML algorithm signiﬁcantly.
Despite this, many de novo design benchmark tasks either lack an accompanying training set or are frequently used with non-standard, modiﬁed training sets (e.g., subsampling, adding unlabelled data).
Unspeciﬁed Evaluation Budgets : Real de novo design problems that require experimental evaluation will be constrained by a limited time and resource budget, which is not reﬂected in most de novo design benchmarks.
Researchers either focus on the setting in which there is an inﬁnite budget or set an arbitrary ﬁnite budget.
This is problematic since the available
number of objective function evaluations can fundamentally change the nature and difﬁculty of an optimization problem.
For example, a small budget may favor exploitative algorithms, whereas a larger budget requires a balance of exploration and exploitation.
The ﬁeld of de novo design would beneﬁt from the adoption of high-quality standardized benchmarks.
These benchmarks should include an explicitly deﬁned objective function, dataset, and evaluation budget.
Furthermore, they ought to closely reﬂect the difﬁculty of de novo design problems.
However, the best benchmark is a priori unclear partly because there are no known systematic evaluations of benchmark candidates.
This work provides a preliminary comparison of different benchmark candidates by running a few established algorithms on many previously proposed benchmarks.
We test 16 variations of each benchmark with different training set sizes and evaluation budgets rather than choosing a single setting as commonly done in the literature.
In total, we performed overexperiments.
We analyze the difﬁculty of each benchmark in detail and provide fresh insights into the merits of each objective function, dataset, and budget combination.
Based on our results, we offer concrete suggestions to researchers and propose directions for the ﬁeld to adopt better benchmarks: (1) ﬁx datasets and budgets for a fair comparison, (2) test multiple datasets and budget settings for each objective, (3) do not use (penalized) logP and QED objectives, and (4) use GuacaMol and DOCKSTRING instead.
Experimental Setup
Starting from a list of algorithms, objective functions, datasets, and budgets, we ran an experiment with every algorithm/objective/dataset/budget combination.
We performed three replicates of each experiment to account for randomness.
Below, we specify the experimental setups.
De novo design algorithms.
In total, we tested ﬁve different algorithms: “Dataset Best”, “Random
ZINC”, Graph GA, SELFIES GA, and GP BO.
“Dataset Best” is a trivial algorithm that performs no queries and returns the best molecule in the training set.
“Random ZINC” ignores the training set and uses the budget to randomly evaluate molecules from the ZINC dataset of ~1 billion purchasable drug-like molecules [Irwin et al., 2012, 2020].
We chose these trivial baselines as a sanity check: if either of these algorithms is competitive, the task must be too easy.
Graph GA and SELFIES
GA are genetic algorithms (GAs) that use graph-based [Jensen, 2019] and string-based [Krenn et al., 2020] representations of molecules, respectively.
We selected these algorithms to represent prototypical exploratory algorithms: if these algorithms are competitive, the task can be solved chieﬂy via exploration.
Gaussian process Bayesian optimization (GP BO)
[Srinivas et al., 2010] is a well-established algorithm known for its capacity for both exploration and exploitation and its good performance in the low-budget regime.
A priori GP BO was expected to be the best-performing algorithm.
Further details on these algorithms are given in Appendix B.
We are aware that many algorithm classes and variants are not represented here: these algorithms were chosen because they are well-established, fast, work well with a wide range of dataset sizes and budgets, and have very few hyperparameters to tune.
Objective functions.
We examined 11 objective functions in total.
Maximization of logP [Wildman and Crippen, 1999], penalized logP
[Gómez-Bombarelli et al., 2018], and QED [Bickerton et al., 2012] were chosen due to their frequent use in previous work, despite their well-documented limitations.
We selected the Celecoxib and Troglitazone rediscovery, Median Molecules 1 and 2, and Osimertinib and Zaleplon MPO ( multi-property objective ) objectives from the 20 goal-directed generation objectives in the GuacaMol benchmark suite [Brown et al., 2019] (chosen to represent the three major task types in their benchmark suite).
Finally, we selected two tasks from the recently published DOCKSTRING study [García-Ortegón et al., 2021]: F2 and selective JAK2.
These tasks both involve proposing drug-like molecules with low binding free energies to a target protein. The objective functions are described in more detail in Appendix C.
Datasets.
In this work, we vary only the size of the dataset and not its distribution.
We choose to examine datasets of size { 10 2 , 10 3 , 10 4 , 10 5 } produced by uniformly subsampling molecules from a larger dataset (without replacement).
For logP, QED, and GuacaMol tasks, the GuacaMol dataset consisting of ~ 10 6 molecules is employed.
For the DOCKSTRING tasks, ~250 000 molecules from the accompanying dataset are used.
Evaluation Budget.
For each experiment, we ﬁxed the number of objective function evaluations, i.e., computational budget.
The list of budgets was: { 10 2 , 10 3 , 10 4 , 10 5 } .
Due to its relatively high computational cost, we omit the 10 5 budget for the DOCKSTRING tasks.
Results
The results are summarized in Figure 1 (see also Table A.1).
First, it can be seen that the dataset size and budget affect the performance of all algorithms.
As one would expect, algorithms generally tend to achieve higher scores with larger datasets and higher budgets.
But there are exceptions: for example, the performance of GP BO with a budget of 10 5 is virtually the same for all dataset sizes on
Celecoxib.
Second, for each objective, the ranking of different algorithms can change depending on the budget and initial dataset size.
For example, for p-logP with an initial dataset size and budget of 10 5 , the order is SELFIES GA > Graph GA > GP BO, while with an initial dataset size and budget 10 3 , the order is reversed.
Third, for a ﬁxed initial dataset and budget, the order can change for different objectives: for example, for an initial dataset size and budget of 10 5 , the order on Zaleplon
MPO is GP BO > Graph GA > SELFIES GA while it is Graph GA > SELFIES GA > GP BO on
Osimertinib MPO.
Overall our ﬁndings indicate that the selected tasks vary signiﬁcantly in difﬁculty and that measuring performance on one task is not a reliable predictor of performance on another one.
We give a brief interpretation of the results for each objective below, with more details in Appendix A. logP and penalized logP are exploited by algorithms that propose large molecules (such as SELFIES
GA).
We recommend against using these objectives.
QED is too easy: performance saturates at 0.948 with even a moderate budget and is also achieved by all baseline algorithms, including random ZINC and dataset best.
Celecoxib and Troglitazone Rediscovery are such that some but not all algorithms achieve the maximum score of 1.0 with a high budget or large initial dataset.
Troglitazone is the more challenging of the two since the perfect score is achieved less often.
Median Molecules 1 and 2 showcase that the performance of algorithms varies signiﬁcantly with dataset size and budget.
Median mols 1 seems more sensitive to dataset size and budget. Osimertinib and Zaleplon MPO are difﬁcult and look qualitatively different: Osimertinib MPO appears to require signiﬁcant exploration, whereas Zaleplon MPO seems to require both signiﬁcant exploration and exploitation. DOCKSTRING tasks seem more exploratory as performance depends weakly on the dataset size, but the strong performance of GP BO suggests that exploration alone is insufﬁcient.
Overall they appear to be qualitatively different from all other GuacaMol objectives.
Conclusions
Based on the results, we have the following recommendations:
Fix datasets and budgets for a fair comparison.
The performance of the baseline algorithms varies considerably for the same objective with different budgets/dataset sizes.
Therefore, to fairly compare algorithms, the same dataset and budget must be used.
Test multiple datasets and budget settings for each objective.
The relative performance of algorithms can depend on the initial dataset and budget.
This dependence can be easily missed if only one setting for the dataset and budget is used.
If it is necessary to choose only one conﬁguration, a principled justiﬁcation should be given.
Do not use (penalized) logP and QED objectives.
Despite previous works decrying how poor these objectives are [Brown et al., 2019, García-Ortegón et al., 2021], they continue to be used.
Our results provide further evidence that these tasks are poor benchmarks.
Use GuacaMol and DOCKSTRING instead.
The GuacaMol benchmarks have objectives of varying difﬁculty, with the MPO objectives being particularly challenging.
The DOCKSTRING objectives are difﬁcult and qualitatively different from GuacaMol tasks.
They have the bonus of being medicinally relevant, albeit computationally expensive.
Our main contribution has been to highlight the importance of specifying the dataset and budget of a benchmark, and we hope that the ﬁeld moves towards standardizing and specifying these in their future work. An expanded analysis of the variance of different algorithms and the diversity of the molecules produced would be an excellent follow-up to the paper.
We hope this paper inspires the community to re-access its commonly used benchmarks.
[Figure]
Graph GA
SELFIES GA
GP BO
[Figure]
A Additional Results
The complete results can be found in Table A.1.
We give a more detailed assessment of the objectives in Appendix A.1.
In Appendix A.2 we answer some additional questions about the results.
A.1 Comments On Each Objective logP and penalized logP.
With a sufﬁcient budget, all non-trivial algorithms signiﬁcantly outperform the best in the dataset.
logP and penalized logP are easy to maximize by adding more (carbon) atoms to a molecule.
Therefore, this objective can efﬁciently be maximized ad inﬁnitum by algorithms.
Graph GA, SELFIES GA, and GP BO do this, and consequently, we see steadily increasing performance with an increasing budget.
Graph GA and GP BO have an internal bias against proposing molecules with high molecular weight.
Therefore, they are outperformed by
SELFIES GA, which does not have such an internal bias.
However, SELFIES GA performs worse at most other tasks that are more demanding.
In conclusion, logP variants are poor proxies for different tasks, and we recommend against their use.
QED.
QED appears to have a global maximum at 0.948, as none of our algorithms and previous works failed to ﬁnd a molecule with a higher score [Guimaraes et al., 2017, Liu et al., 2018, You et al., 2018, Jin et al., 2018, Zhou et al., 2019].
All algorithms reached this value with a high budget (the trivial Dataset Best and Random ZINC baselines even with moderate budgets).
This suggests that the task is too easy to determine the relative performance of algorithms; at best, it could be used as a sanity check or as a benchmark in the extremely low-budget setting.
Overall we recommend against using this objective.
Celecoxib and Troglitazone Rediscovery.
These objectives are challenging, as the trivial baselines and SELFIES GA do not achieve the theoretical maximum of 1.0.
However, Graph GA and GP BO obtain good results with a moderate budget and a large dataset.
Celecoxib seems to be slightly easier than Troglitazone as the highest possible score is achieved with multiple settings.
These objectives could be employed in the low data or low budget settings or as a toy objective.
Median Molecules 1 and 2.
The performance of all algorithms varies with dataset size and budget.
Median Molecules 1 seems more difﬁcult as good performance is achieved only with a high budget and a large dataset.
In contrast, the performance on the Median Molecules 2 task can be improved with just a large dataset (at least with GP BO).
For small budgets, the best molecule in the dataset is competitive.
Osimertinib and Zaleplon MPO.
The plots for Osimertinib MPO resemble those of logP, with performance depending more strongly on the budget than on the dataset size.
This suggests that the task requires more exploration than exploitation, and that the GuacaMol dataset does not contain many molecules close to the optimum.
The fact that both genetic algorithms outperform GP BO bolsters this claim since these algorithms are more exploratory.
For Zaleplon MPO, the performance of GP BO increases signiﬁcantly with dataset size, whereas for the other methods, there is a comparatively small increase.
This behavior is not seen with any other objective function, suggesting that Zaleplon MPO has unique difﬁculties not present in the other benchmarks.
GP BO also signiﬁcantly outperforms the genetic algorithms, suggesting that exploitation is necessary to achieve competitive results.
Since GP
BO achieved the best score in the maximum data and budget setting, we conclude that Zaleplon MPO requires signiﬁcant exploration and exploitation.
This appears to be the most challenging GuacaMol task out of the ones surveyed.
DOCKSTRING tasks.
First, all algorithms see an increased performance with an increasing budget, suggesting that the global maximum is not in or near the dataset.
Secondly, performance varies less signiﬁcantly with dataset size than other tasks, suggesting that the dataset does not contain many points near the optimum.
Further, signiﬁcant exploration is required for good performance.
Unlike Osimertinib MPO, GP BO signiﬁcantly outperforms both genetic algorithms with a high budget, showing that exploitation is necessary (or that random exploration is less likely to succeed).
Also, unlike the GuacaMol tasks, SELFIES GA fails to outperform the best in the dataset in almost every setting.
These tasks are qualitatively different from any other GuacaMol tasks and would be good benchmarks for novel algorithms in various settings, mainly owing to their medicinal relevance.
in Figure
The abbreviated task names are penalized logP (p-logP)
Celecoxib rediscovery ( rediscovery
[Table]
A.2 Miscellaneous Questions about the results Were the same datasets used for all runs?
For each dataset size, we randomly sub-sampled ten datasets from the total dataset. For the replications of individual experiments, we used different datasets to account for variation in performance due to which dataset was chosen (therefore, three datasets were used for each experiment). However, the same datasets were used for each different algorithm so that they are directly comparable.
To give a concrete example, for the task “Celecoxib rediscovery” with dataset size 10 3 , we had a pool of 10 different datasets (#1–#10) sub-sampled from the larger GuacaMol training set of size ≈ 1 . 5 × 10 6 .
Each experiment (e.g., SELFIES GA with budget = 10 4 , GP BO with budget = 10 2 ) was run three times, the ﬁrst using dataset #1, the second using dataset #2, and the third using dataset #3.
Non-monotonicity in performance: is this odd or unexpected?
There are various nonmonotonic trends that can be observed in Figure 1.
For example, in SELFIES GA for Median
Molecules 2 with a dataset of size 10 4 , the performance is slightly worse with a budget of 10than 10 3 .
Another example is in GP-BO for the JAK2-LCK task with a budget of 10 4 , where the performance with initial dataset size 10 5 is worse than with an initial dataset size of 10 4 or 10 3 .
There are several possible explanations for this:
• Randomness: all the optimization procedures tested are stochastic, so there is inherently some variation between different runs.
With only three trials for each experiment, the empirical median is a rough noisy estimate of the true median, and therefore some truly monotonic trends may appear to be non-monotonic.
• Local optima: the chance of getting stuck in a local optimum does not have a clear dependence on dataset size.
For example, with a small dataset, an algorithm might get stuck exploiting the optimum closest to the initial dataset.
Alternatively, a large dataset may contain more local optima, which could all take time to be explored before the algorithm is able to escape.
• Explore-exploit behavior: the algorithms tested combine elements of “exploration” and “exploitation”.
Depending on the size and composition of the known dataset, the balance of exploration and exploitation in a different algorithm can change.
While we are unsure exactly what the cause of each monotonic trend is, we conjecture that for the two genetic algorithms, the most important factor is randomness, while for the GP-BO algorithm, it is getting stuck in local optima.
B Experimental Details
The implementation of Graph GA was taken directly from García-Ortegón et al.
The implementation of GP BO was a basic Bayesian optimization loop using a GP with the Tanimoto kernel [Tanimoto, 1958] and an upper conﬁdence bound acquisition function [Srinivas et al., 2010].
The implementation was based on García-Ortegón et al.
but was modiﬁed to use a larger batch size for the budget of 10 5 .
Further, it was modiﬁed to employ a subset of the data at all times rather than simply starting with a subset of the data.
The SELFIES GA implementation was based heavily on the Graph GA, except that the mutation step used the SELFIES
[Krenn et al., 2020] genetic algorithm mutation function from Nigam et al.
Code is available at https://github.com/AustinT/ai4sci-2021-denovo-benchmarks/ .
C Explanation of objectives
A brief explanation of all the objectives is given below.
For more details, see the original references for logP [Wildman and Crippen, 1999, Gómez-Bombarelli et al., 2018], QED [Bickerton et al., 2012],
GuacaMol [Brown et al., 2019], and DOCKSTRING [García-Ortegón et al., 2021] objectives.
• logP and penalized logP: a measure of solubility in oil; approximated as a sum over atoms where each atom type has either a positive or negative contribution to the total.
penalized logP adds two additional terms (synthetic accessibility and a penalty for large rings) and reweights all the terms.
• QED: a measure of drug-likeness between zero and one.
It penalizes molecules for violating Lipinski’s rules on molecular weight, logP, and number of hydrogen bond donors and acceptors.
It also penalizes certain moieties.
• Celecoxib and Troglitazone rediscovery (GuacaMol): similarity to the known drug molecules Celecoxib and Troglitazone, respectively, as measured by Morgan-2 ﬁngerprints.
• Median Molecules 1 and 2 (GuacaMol): the geometric mean of ﬁngerprint similarities of two relatively different molecules.
For Median Molecules 1, these molecules are camphor and menthol.
For Median Molecules 2, these molecules are tadalaﬁl and sildenaﬁl.
• Osimertinib MPO (GuacaMol): a complicated function rewarding molecules for be similar (but not too similar) to the drug Osimertinib and for having the logP and TPSA in a drug-like range.
• Zaleplon MPO (GuacaMol): function which rewards molecules similar to the drug zaleplon and whose chemical formula is C 19 H 17 N 3 O 2 (different than zaleplon’s actual chemical formula).
• F2 ( DOCKSTRING ): docking score against the F2 protein (Thrombin), which is involved in blood clotting, with a penalty against molecules with low QED.
• Selective JAK2 ( DOCKSTRING ): a complicated function rewarding molecules with strong binding to JAK2 (Janus kinase 2, often mutated in cancers) but weak binding to KIT (tyrosine-protein kinase, also often mutated in cancers), and also a high QED.
It is motivated by the pharmaceutical desire to ﬁnd drugs that bind selectively to a speciﬁc protein.