Bias in the Benchmark: Systematic experimental errors in bioactivity databases confound multi-task and meta-learning algorithms
Abstract
There is considerable interest in employing deep learning algorithms to predict pharmaceutically relevant properties of small molecules.
To overcome the issues inherent in this low-data regime, researchers are increasingly exploring multi-task and meta-learning algorithms that leverage sets of related biochemical and toxicological assays to learn robust and generalisable representations.
However, we show that the data from which commonly used multi-task benchmarks are derived often exhibits systematic experimental errors that lead to confounding statistical dependencies across tasks.
Representation learning models that aim to acquire an inductive bias in this domain risk compounding these biases and may overfit to patterns that are counterproductive to many downstream applications of interest.
We investigate to what extent these issues are reflected in the molecular embeddings learned by multitask graph neural networks and discuss methods to address this pathology.
1. Introduction
The impressive performance of modern deep learning algorithms in traditional machine learning domains such as computer vision and natural language processing has led to a resurgence of interest in applying them to various problems throughout the drug discovery pipeline.
While many parts of the pre-clinical drug discovery process stand to benefit from robust computational models, the tasks of predicting the bioactivity or toxicity of a molecule from its structure have emerged as one of the most popular benchmarks for algorithmic progress in this area (Wu et al., 2018; Mayr et al., 2018).
One of the fundamental practical issues of adapting existing and developing novel approaches to this problem is that the size of available high-quality labelled datasets is orders of magnitude below that of the well-known benchmarks to which much of the recent progress in e.g. computer vision has been ascribed (Sun et al., 2017).
This limitation stems from the inherently experimental nature of medicinal chemistry, and while high-throughput screening (HTS) technologies are steadily improving, it is unlikely that this bottleneck will be overcome in the near future.
To train models that are viable in this low-data regime, a considerable amount of recent work has investigated the ability of multi-task and meta-learning algorithms to leverage a collection of related datasets (e.g. Ramsundar et al.
; Lenselink et al. ; Mayr et al.
; Nguyen et al.
; Stanley et al.
The rationale behind these approaches is that requiring a model to perform well on a set of related but distinct tasks leads to more powerful and generalisable representations, mediated by both the increased quantity of training data and the implicit regularisation of optimising multiple predictive performances.
This objective of attending to underlying causal factors that are shared across tasks, while de-emphasising task-specific noise, is referred to as learning a domain-specific inductive bias (Caruana, 1997).
However, this approach is clearly counter-productive if the set of tasks from which an inductive bias is derived exhibits statistical dependencies that are adversarial to a particular downstream application of interest.
For example, one of the main limitations of high-throughput bioactivity and toxicity screens is their tendency to produce large numbers of reproducible false positives.
These well-known artefacts often stem from compound-dependent interference with the assay system and can be responsible for up to 95% of ostensibly active compounds (Thorne et al., 2010).
As many of the mechanisms that are known to produce these false positives act orthogonally to any specific biochemical interaction of interest, it is plausible to assume that they persist across different assays, which is indeed what is observed in practice (Baell & Holloway, 2010; M Nissink & Blackburn, 2014;
Schorpp et al., 2014)
The prevalence of these falsely active compounds and their recurrence across biologically unrelated systems presents a challenge for many multi-task and meta-learning algorithms.
If a substantial proportion of positive labels can be consistently attributed to a small set of assay-interfering substructures, this may present a more attractive inductive bias than learning complex biochemical interactions, especially if the representational capacity of a model is restricted by further regularisation, as is common in low-data domains.
In the following, we will review how bioactivity data is measured and highlight different forms of assay interference (Section 2).
We then demonstrate that the resulting biases are prevalent many popular bioactivity benchmarks and influence the representations learned by multi-task graph neural networks (Sections 3 and 4).
Finally, we propose approaches to address this issue (Section 5).
2. Measuring and Mismeasuring Bioactivity
Identifying molecules that have a strong effect on a given target is challenging, as usually only very few compounds elicit a desired response.
This is compounded by the fact that chemical space is vast, with estimates for the number of unique drug-like molecules ranging from 10 20 to 10(Bohacek et al., 1996; Ertl, 2003; Polishchuk et al., 2013).
To increase the coverage with which molecules can be investigated, drug discovery research makes use of highthroughput screens (HTS) to identify promising lead candidates for extensive follow-up studies.
These usually consist of a large molecular library being passed in an automated fashion through a miniaturised assay system that translates the physiological effect of a molecule into an easily measurable readout.
As in any chemical system, the magnitude of the effect that a compound exerts is dependent on its concentration, and the objective of most screening campaigns is to find molecules that show a strong biological effect at a minimal dose.
A common approach to identifying such compounds is to first assay the entire compound library at a single concentration, and then select the molecules with the strongest response to be re-measured in a dilution series.
A logistic model is then fit to this dose-response data and used to derive quantitative measures of the binding dynamics, such as the widely-used half-maximal active concentration 1 .
One common issue with HTS assay systems is that the biological target of interest is not the only or even the main component that molecules can interact with.
Different classes of substructural motifs can elicit a wide variety of behaviours that are orthogonal to the physiological interaction being investigated, but nevertheless result in a positive readout.
Examples include compound aggregation (McGovern et al., 2002; Ryan et al., 2003; Feng et al., 2007), reactivity towards the protein target (Rishton, 1997), and interference with the readout method (Baell & Holloway, 2010; Baell &
Nissink, 2018).
As these confounding interactions are not experimental noise, but reproducible, concentration-dependent, and often platform-independent positive readouts, it is generally difficult to differentiate them from truly active compounds without extensive validation experiments (termed counterscreens).
Researchers have extensively examined historic
HTS data and collated lists of molecular substructures that are enriched in compounds with high frequencies of hits, exemplified by the well-known library of Pan-Assay INterference compoundS, or PAINS (Baell & Holloway, 2010).
[Figure]
And while medicinal chemists have grown cognisant of seemingly active compounds containing known assayinterfering substructures, they remain prevalent in popular bioactivity databases and published research (Dahlin &
Walters, 2016).
Moreover, many unidentified or less extensively characterised substructural motifs are bound to exist in high-throughput screening datasets, further complicating analysis.
As outlined in Section 1, representations learned by multitask and meta-learning models trained on datasets that are populated with these problematic compound sets risk favouring this more accessible inductive bias over the challenging and complex biochemical interactions that are of practical relevance.
3. Bias in Bioactivity Benchmarks
To quantify the prevalence of this issue in multi-task bioactivity prediction datasets, we investigated the extent to which the hit frequency of a compound across screens can predict binding in any individual screen with a very simple model.
Consider the label matrix Y ∈ { 0 , 1 , – } | M |×| A | of a multi-task binary classification problem, denoting whether a molecule m
i ∈ M = { m 1 , m 2 , . .
, m L } was inactive, active, or not measured in an assay
a j ∈ A
=
{ a 1 , a 2 , . .
. , a K } .
For all molecules that were measured in a given assay a j
[Equation]
we calculate the frequency of hits across all other assays they were measured in
[Equation]
and use these hit frequencies to calculate the areas under the receiver operator characteristic (AUC-ROC) and the precision recall curve (AUC-PRC) with respect to the labels in a j . Molecules that are only measured in a single assay are assigned the global hit frequency across Y 2 .
The summary of this characterisation is shown in Table 1, where the metrics are averaged across all assays and compared against the expected performance of a random classifier using the label ratio of each a j .
If the assays in A investigate reasonably distinct biological systems, as is the case in public repositories such as PubChem (Kim et al., 2021) and ChEMBL (Gaulton et al., 2017), this approach is not expected to perform significantly better than a random classifier.
These databases cover hundreds of different targets spanning many physiological functionalities and scales, meaning that any consistent activity across screens is much more likely to stem from target-independent interference mechanisms than selective biochemical interactions.
To characterise this behaviour, five multi-task benchmarks were retrieved from their respective sources and the MoleculeNet repository (Wu et al., 2018), including:
C H EMBL - a preprocessed subset of the ChEMBL database (Mayr et al., 2018), designed for multi-task bioactivity prediction; MUV- an extensively preprocessed subset of the PubChem database (Rohrer & Baumann, 2009), originally designed for virtual screening; PCBA - a large, minimally preprocessed subset of the PubChem database (Ramsundar et al., 2015), designed for multitask bioactivity prediction; and T OX 21/T OX C AST - a collection of assays measuring binding to targets involved in known adverse physiological reactions (Dix et al., 2007).
The degree to which the issue of target-independent assay interference is addressed varies between benchmarks.
[Table] While the authors of the PCBA dataset acknowledge the possibility of multi-task models focussing on interfering substructures, they argue that the predictive performance of networks trained on cleaner data alleviates this concern.
The authors of the MUV dataset go further and apply extensive preprocessing steps, including the removal of all compounds with a hit rate over 26%.
The curators of T OX C AST pursue an even more rigorous approach and perform extensive experimental counter-screens for each assay.
However, in the form that this dataset is currently distributed and used these counter-screens are simply treated as additional predictive tasks, strongly amplifying the adverse incentive presented by assay-interfering compounds.
The results of our experiment in Table 1 closely mirror these considerations.
The datasets that have not been preprocessed to remove potential artefacts (C H EMBL, PCBA, T OX 21, and T OX C AST ) show a substantial correlation between the readouts of different assays.
The only dataset on which the performance is close to random is MUV, where many potentially interfering compounds are filtered out.
4. Impact on Learned Representations
To investigate to what extent these statistical dependencies are reflected in the representations that models learn, we examined the molecular embeddings that a multi-head graph neural network derives from the C H EMBL dataset.
Specifically, we used the graph isomorphism network (Xu et al., 2018) architecture introduced in Hu et al.
(2019), consisting of a representation-learning network ψ : G → R d that maps a molecular graph G to a continuous embedding in R d , followed by a multi-head linear layer ϕ : R d → R | A | to model bioactivity labels.
In the following experiments, we compare the molecular representations generated by a random initialisation 3 of ψ to those that were optimised to perform well in the supervised multi-task setting by (Hu et al., 2019).
Our hypothesis is that the strong predictive performance of a frequency of hits regression model demonstrated above leads to representations that encode molecular promiscuity and propensity to target-independent assay interference, rather than modelling the biochemical concepts underlying target-specific interactions.
To quantify this, we replaced the last layer of the multi-task[-
 In contrast to other deep learning models, it is well-established that graph neural networks can already extract highly useful rep- resentations in their randomly initialised & Welling,-] network with a single-head regression output ϕ ′ :
R d → R , keeping ψ fixed, and trained this new predictive layer on hit frequencies across the C H EMBL dataset.
As expected, we observe that a linear model fit on top of the pretrained embeddings is able to predict molecular hit frequencies much better than one that is trained on the naive embeddings, summarised by Pearson correlations of 0.43 and 0.
respectively.
A complimentary approach to testing this hypothesis is to examine the topology of the learned representation space.
Rohrer & Baumann (2009) outline the method of refined nearest neighbour analysis, which can be used to describe how evenly two classes of molecules are distributed in chemical space.
We adapt this method to quantify how clearly compounds containing well-known assay-interfering molecular substructures are separated from compounds that do not and use the resulting metric to compare the pretrained and na ¨ ıve embedding spaces.
First, we annotate all compounds from the C H EMBL benchmark containing one or more of the substructures enumerated in the PAINS library (Baell & Holloway, 2010).
As a sanity check, we compare the distribution of hit frequencies between these and the remaining compounds and find that flagged molecules have a significantly 4 higher hit rate than molecules without PAINS substructures.
Next, we define
G ( d i ) as the proportion of flagged compounds for which the distance to the nearest other flagged compound is less than d i and the function F ( d i ) as the proportion of PAINS-free compounds for which the distance to the nearest flagged compound is less than d i .
The resulting distributional metric
[Equation]
− G ( d i )) captures the cumulative difference between F and G over a set of distance thresholds { d i }
N d i =1 .
Intuitively, if F ( d i ) is consistently lower than G ( d i ) over a range of different distance thresholds, i.e. when S < 0 , molecules that contain PAINS substructures are clustered in the embedding space and are thus more easily distinguishable from molecules that do not.
We follow Rohrer & Baumann (2009) and generate the distance threshold sets by creating N d = 500 evenly spaced splits in [0 , 3 d m ] , where d m is the median nearest neighbour distance.
Consistent with the findings from the previous experiment, the representation space of the trained network exhibits a significantly greater separability of PAINS-containing compounds ( S = − 0 . 39 ) than the representation space derived from a random initialisation ( S = − 0 . 29 ).
While a more thorough investigation of different models in different settings is necessary, these results indicate that multi-task datasets on which significant discriminative performance can be achieved by recognising assay-interfering substructures are able to induce representations that reflect this shortcut.
And while these representations may perform well on the benchmark in question, they are inadequate for many downstream applications, as for example a generative model built on them would preferentially generate assay-interfering compounds and not molecules with actual physiological activity.
5. Conclusions and Future Work
Systematic experimental errors that cause assayindependent false positive readouts in biochemical and toxicological high-throughput screens are present in many popular multi-task benchmarks.
The resulting statistical dependencies lead to a high correlation between tasks and can compel multi-task and meta-learning algorithms to focus on predicting hit frequencies, leading to representations that are counterproductive to many impactful downstream applications of interest.
The most robust way to address this issue is to make use of experimental information from counter-screens and remove active compounds that are likely assay-interfering or nonselective.
Several instances of high-throughput screens with associated validation assays exist (see e.g. Butkiewicz et al. and could be compiled into a collection of highquality data that is used as a clean test set.
However, the vast majority of screening data in repositories such as ChEMBL or PubChem is not linked to assay-specific counter-screens that could be used to filter out potential false positives.
To still recognise problematic compounds in these screens, it may be beneficial to examine the associated measurement meta-data, as the MUV dataset shows that even highly empirical rules-of-thumb can already significantly improve the quality of a dataset.
Going even further, the large quantities of meta-data in repositories such as PubChem could present an opportunity to systematically develop more sophisticated de-biasing and de-confounding techniques.
Conclusions from these techniques could then in turn be used to process new single- and multi-task datasets and increase the efficacy with which molecular property prediction models can learn from publicly available bioactivity data.