Improving Hit-ﬁnding: Multilabel Neural Architecture with DEL
Abstract
DNA-encoded library (DEL) data, often with millions of data points, enables large deep learning models to make real contributions in the drug discovery process (e.g., hit-ﬁnding).
One recent success method of modeling DEL data [1], GCNN [2] multi-class model, requires domain experts to create mutually exclusive classiﬁcation labels from multiple selection readouts of DEL data, which is not always an optimal formulation.
In this work, we designed a multi-label architecture that directly models each selection data to eliminate the corresponding dependency on human expertise.
We selected effective choices for key modeling components such as label reduction scheme from in silico evaluation.
To assess its performance in real-world drug discovery settings, we further carried out prospective wet lab testing where the multi-label model shows consistent improvement in hit-rate (percentage of hits in a proposed molecule list) over the current state-of-the-art multi-class model.
Introduction
One of the major barriers in applying machine learning, especially deep learning, to drug discovery problems is limited data.
The recent advent of DNA-encoded libraries (DELs) with their massive data size opens many new opportunities [3] [4] [5].
Training large deep learning molecular property models becomes possible and applying them to key stages (e g., hit-ﬁnding, hit to lead) of the drug discovery process has shown great promises [1].
The readout of an experimental DEL selection is DNA sequence counts [3], which are commonly aggregated into disynthon representations to calculate enrichment scores with good signal-to-noise properties [1].
Besides the primary DEL selection experiment (measuring binding signal called “Target Enr” when protein target is present, i.e., on-target binding experiment), control selection (measuring binding signal called “NTC Enr” when no protein target is present, i.e., No-Target Control binding experiment) and additional counter-selections could be run to further reduce noise, for example, inclusion of a known competitive inhibitor (measuring binding signal called “Competitor Enr” when protein target is competitively inhibited)
In McCloskey et al.
’s
work [1], data from those multiple DEL selections are combined to create mutually exclusive labels corresponding to different experimental outcomes.
These labels represent logical combinations (e.g. AND/OR) of enrichment scores from the multiple experimental conditions as well as external side information such as hit frequency across targets denoted as “Target ratio” (to identify promiscuous compounds that are unlikely to have speciﬁc interactions with the target).
The label derivation can be often summarized in a decision tree, which we call a column scheme .
See an example in Figure 1.
A classiﬁcation model can then be readily built with the derived classes.
One major problem with this approach is that in reality a molecule being categorized as one class can still be of another class (e.g., a promiscuous compound can still be a target competitive hit).
Additionally, using labels derived from multiple experiments, each with their own criteria and thresholds, requires the model to learn a complicated latent structure for the human-crafted labels (ﬁxed at training time, see Figure 1).
Our key insight is that the separate DEL experimental outcomes can be treated independently.
In this way, the label naturally matches with the physical meaning of each selection experiment—the resulting model makes a prediction for each experimental outcome and these predictions can be ﬂexibly combined as needed in downstream applications.
We summarize our contributions in this paper as follows:
• we design and implement key components of a multi-label neural architecture that models DEL data more naturally (Figure 2), • we conduct in silico retrospective testing experiments and demonstrate this philosophy of modeling each DEL selection experiment independently can bring better performance than the multi-class architecture, • to examine performance in real drug-discovery settings, we carry out wet lab prospective testing experiments [6] for two protein targets, which showed superior performance of multi-label models in hit-ﬁnding applications.
Results
In order to compare hit-ﬁnding performance with the current state-of-the-art GCNN multi-class model [1], we selected two relatively hard protein targets: Tyrosine-protein kinase (c-KIT) and
Estrogen Receptor Alpha (ERa).
In [1], c-KIT and ERa show relatively low hit rates (9.7% and 18.8% respectively at concentration of 10uM), giving this study more room to improve.
We trained two types of graph neural networks to enable direct comparison: GCNN multi-class model used in [1] and GCNN multi-label model developed in this study (see Figure 2). [Table]
2.1 In silico evaluation
As a ﬁrst step [Table] we carried out in silico retrospective evaluation of our models against internal inhibitor datasets.
Speciﬁcally [Table] we have ﬁve test datasets for ERa and two for c-KIT.
We monitor a metric that is closely related to hit-rate: the number of active molecules among the top 100 highest scoring molecules in the test set (actives@100).
Table 1 shows the results.
Despite the slight improvement, the GCNN multi-label models outperforms its multi-class counterpart for most of the test sets (6 out of 7)
2.2 Wet-lab results
Improving hit-ﬁnding performance The training data follows the same preprocessing step with disynthon aggregation (Methods section of [1]), which is mainly to denoise the raw DNA-sequencing count data.
During prospective experimental testing, each model type proposes a list of ∼molecules from the same commercially purchasable library (Mcule instock library [7] was used in this study) and the percentage of hits is measured in wet lab.
We limit human intervention in the molecule list proposal by automating diversity selection and structural ﬁltering into a streamlined pipeline, so that the hit-rate difference should be mostly explained by model type difference.
Simpliﬁed from a two-step prospective testing approach [1], we use single-point inhibition assays evaluated at the molecule concentration of 10uM in this study. [Figure]
[Figure] For both of the c-KIT and ERa protein targets, GCNN multilabel model outperformed GCNN multiclass model at three different inhibition cut-off percentages (see Figure 3).
Speciﬁcally for c-KIT target, at 30% inhibition (enzyme remaining activity = 70%), multilabel model achieved 26.2% hit rate whereas multiclass model had 20.9%; at 30% inhibition, multilabel model achieved 20.1% hit rate whereas multiclass model had 17.4%.
The hit rates achieved by multiclass model in [1] are also included as additional reference; the difference between multiclass model in this study and multiclass model in [1] come from recent improvements in model hyper parameters, difference in commercial libraries to select molecule list from, difference in diversity and ﬁltering strategies.
Pushing hits to high score region Why does the GCNN multilabel model achieve a better hitrate than the multiclass model?
We notice that, via an internal ERa inhibitor dataset (test1) as a demonstrating example, GCNN multiclass model has placed many active molecules in the middle range of the score spectrum.
This could be caused by unrealistic assumptions made in creating the mutually exclusive labels.
On the other hand, GCNN multilabel model tends to score the active molecules (true inhibitors) more towards the high end compared with the GCNN multiclass model (see Figure 4a).
Figure 4b suggests GCNN multilabel model has an improved calibration especially in the middle score region.
This helps enrich the actives in the high score region.
Selecting key components
There are several key components that are essential for multilabel neural architecture to work properly.
Below we disclose the our detailed choices in two important components: data sampling, label reduction scheme
Data sampling strategy We design the data sampling as follows.
Each selection experiment label has a wide range of enrichment scores.
We categorize each label into high enrichment and low enrichment types and feed the model during training with equal chance of high vs low enrichment data types.
Furthermore, if we have 3 labels (i.e., 3 selection experiments), then the number of all combinatorial types is 8.
We make sure each training data batch has balanced example counts between all these types.
Label reduction scheme Coming out of the GCNN multilabel model are multiple prediction scores (one per selection experiment), so how to effectively combine them into a single prediction (i.e., reduced label) and use it for checkpoint selection as well as downstream tasks (e.g., propose molecule list to ﬁnd hits) are crucial for a successful multilabel model.
Through in-silico evaluation, the ﬁnal label reduction scheme used in this study is:
[Equation]
Conclusion
In this study, we have presented a simple multilabel architecture for DEL data modeling, which allows models to learn more naturally with the DEL data at hand than the current state-of-the-art approach.
An equally important aspect is that it helps eliminate the dependency on human-crafted column scheme.
This simple multilabel architecture not only shows improvements in retrospective test datasets, but also, through web-lab prospective testing, brings real improvements in hit-ﬁnding use-case.
Although the fundamental reason behind the effectiveness of multilabel models is yet to uncover, we observe that its improved calibration tends to score those active compounds towards higher region than the GCNN baseline model does.