Original: I MPROVING THE ASSESSMENT OF DEEP LEARNING MODELS IN THE CONTEXT OF DRUG - TARGET INTERACTION PREDICTION
Revised: I MPROVING THE ASSESSMENT OF DEEP LEARNING MODELS IN THE CONTEXT OF DRUG - TARGET INTERACTION PREDICTION

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine Learning techniques have been widely adopted to predict drug-target interactions, a central area of research in early drug discovery.
Revised: Machine Learning techniques have been widely adopted to predict drug-target interactions, a central area of research in early drug discovery.

Original: These techniques have shown promising results on various benchmarks although they tend to suffer from poor generalization.
Revised: These techniques have shown promising results on various benchmarks although they tend to suffer from poor generalization.

Original: This is typically related to very sparse and nonuniform datasets available, which limits the applicability domain of Machine Learning techniques.
Revised: This is typically related to very sparse and nonuniform datasets available, which limits the applicability domain of machine learning techniques.

Original: Moreover, widespread approaches to split datasets (into training and test sets) treat a drug-target interaction as an independent entities, when in reality the drug and target involved may take part in other interactions, breaking apart the assumption of independence.
Revised: Moreover, widespread approaches to split datasets (into training and test sets) treat a drug-target interaction as an independent entities, when in reality the drug and target involved may take part in other interactions, breaking apart the assumption of independence.

Original: We observe that this leads to overly optimistic test results and poor generalization of out-of-distribution samples for various state-ofthe-art sequence-based Machine Learning models for drug-target prediction.
Revised: We observe that this leads to overly optimistic test results and poor generalization of out-of-distribution samples for various state-ofthe-art sequence-based machine learning models for drug-target prediction.

Original: We show that previous approaches to reduce bias in binding datasets focus on drug or target information only and, thus, lead to similar pitfalls.
Revised: We show that previous approaches to reduce bias in binding datasets focus on drug or target information only and, thus, lead to similar pitfalls.

Original: Finally, we propose a minimum viable solution to evaluate the generalization capability of a Machine Learning model based on the systematic separation of test samples with respect to drugs and targets in the training set, thus discerning the three out-of-distribution scenarios seen at test time: (1) drug or (2) target present in the training set, or neither (3).
Revised: Finally, we propose a minimum viable solution to evaluate the generalization capability of a machine learning model based on the systematic separation of test samples with respect to drugs and targets in the training set, thus discerning the three out-of-distribution scenarios seen at test time: (1) drug or (2) target present in the training set, or (3) neither.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently developed Deep Learning methods to predict drug-target interactions (DTI) hold great potential for drug discovery (Bagherian et al., 2021).
Revised: Recently developed deep learning methods to predict drug-target interactions (DTI) hold great potential for drug discovery (Bagherian et al., 2021).

Original: In spite of the interest sparked both in academia and industry towards these methods, the lack of standard pipelines or benchmarking criteria discourages adoption and further developments.
Revised: In spite of the interest sparked both in academia and industry towards these methods, the lack of standard pipelines or benchmarking criteria discourages adoption and further developments.

Original: Particularly, as we show here, it is important to assess Deep Learning models beyond usual predictive measures and increase scrutiny on what the models are learning.
Revised: Particularly, as we show here, it is important to assess deep learning models beyond usual predictive measures and increase scrutiny on what the models are learning.

Original: Data quantity and quality play a fundamental role in developing and applying Deep Learning models successfully.
Revised: Data quantity and quality play a fundamental role in developing and applying deep learning models successfully.

Original: In the case of DTI, multiple databases are publicly available and focus on different characteristics of the data, such as: quality of three-dimensional structure (Binding MOAD (Smith et al., 2019)), manual curation of annotations (PDBbind (Liu et al., 2015)), or sparsity, i.e. limiting the number of drugs and targets (ExcapeDB (Sun et al., 2017)).
Revised: In the case of DTI, multiple databases are publicly available and focus on different characteristics of the data, such as: quality of three-dimensional structure (Binding MOAD (Smith et al., 2019)), manual curation of annotations (PDBbind (Liu et al., 2015)), or sparsity, i.e. limiting the number of drugs and targets (ExcapeDB (Sun et al., 2017)).

Original: Here, we perform an independent benchmark of DTI predictors based on BindingDB (Liu et al., 2007), a database collecting interactions from scientiﬁc articles and patents, which has already been used to create DTI benchmark datasets by others (Yingkai Gao et al., 2018; Karimi et al., 2019).
Revised: Here, we perform an independent benchmark of DTI predictors based on BindingDB (Liu et al., 2007), a database collecting interactions from scientiﬁc articles and patents, which has already been used to create DTI benchmark datasets by others (Yingkai Gao et al., 2018; Karimi et al., 2019).

Original: Beyond the speciﬁcs of each dataset, a crucial aspect for developing and assessing a DTI predictor is how to divide or split available data into testing and training sets, while preserving a balanced representation of the interaction space.
Revised: Beyond the speciﬁcs of each dataset, a crucial aspect for developing and assessing a DTI predictor is how to divide or split available data into testing and training sets, while preserving a balanced representation of the interaction space.

Original: This has been demonstrated to be particularly challenging when using chemical data for Machine Learning, given the amplitude of chemical space and the inherent biases present in the sparse and nonuniform DTI benchmark datasets available (Sieg et al., 2019).
Revised: This has been demonstrated to be particularly challenging when using chemical data for machine learning, given the amplitude of chemical space and the inherent biases present in the sparse and nonuniform DTI benchmark datasets available (Sieg et al., 2019).

Original: Moreover, Deep Learning models are capable of easily ﬁtting nuances observed in the training set, i.e. from technical noise to random labeling (Zhang et al., 2021).
Revised: Moreover, deep learning models are capable of easily ﬁtting nuances observed in the training set, i.e. from technical noise to random labeling (Zhang et al., 2021).

Original: Although some approaches have been deployed for evaluating the redundancy between training and test sets of drug-based (Wallach & Heifets, 2018), and target-based (Urban et al., 2020) models, none of these approaches is directly applicable for DTI models.
Revised: Although some approaches have been deployed for evaluating the redundancy between training and test sets of drug-based (Wallach & Heifets, 2018), and target-based (Urban et al., 2020) models, none of these approaches is directly applicable for DTI models.

Original: Thus, state-of-the-art DTI predictors may learn biases observed in DTI benchmark datasets instead of chemical and physical features characterizing potential interactions.
Revised: Thus, state-of-the-art DTI predictors may learn biases observed in DTI benchmark datasets instead of chemical and physical features characterizing potential interactions.

Original: In this work, we benchmark three recently developed DTI methods and observe competitive results on two previously deﬁned benchmark datasets.
Revised: In this work, we benchmark three recently developed DTI methods and observe competitive results on two previously deﬁned benchmark datasets.

Original: However, the models show relatively poor generalization on four Out Of Distribution (OOD) scenarios.
Revised: However, the models show relatively poor generalization on four Out Of Distribution (OOD) scenarios.

Original: Intuitively, a DTI Deep Learning model will generalize to OOD scenarios by learning fundamental chemical and physical features that govern the interaction between a drug and a target at the molecular level.
Revised: Intuitively, a DTI deep learning model will generalize to OOD scenarios by learning fundamental chemical and physical features that govern the interaction between a drug and a target at the molecular level.

Original: To further understand the generalization challenge, we investigated if considering interactions as independent entities, as done  in thesepreviously deﬁned benchmark datasets, may constitute a potential source of information leakage which, in turn, may hinder the learning and evaluation of DTI models.
Revised: To further understand the generalization challenge, we investigated if considering interactions as independent entities, as done previously when splitting the training and test samples in these deﬁned benchmark datasets, may constitute a potential source of information leakage which, in turn, may hinder the learning and evaluation of DTI predictors.

Original: With a baseline analysis, we show that the association of DTIs in these benchmark datasets by drug or target alone contains information and predictive power, limiting the learning process.
Revised: With a baseline analysis, we show that the association of DTI in these benchmark datasets by drug or target alone contains information and predictive power and, thus, constitutes a case of information leakage.

Original: To alleviate this issue, we propose a minimum viable solution for measuring the generalization capability of a DTI predictor, i.e. discerning the three OOD scenarios seen at test time: (1) drug or (2) target present in the training set, or neither (3).
Revised: To improve the assessment of DTI predictors, we propose a minimum viable solution for measuring their generalization capability, i.e. discerning the three OOD scenarios seen at test time: (1) drug or (2) target present in the training set, or neither (3).

Revised: Our solution strengthens the assessment of DTI predictors evaluating empirically their ability to generalize to OOD scenarios, without altering the training set, while facilitating the identiﬁcation of potential sources of bias in benchmark datasets.

Original: To summarize, we show that considering DTIs as independent entities introduces potential sampling biases which may hinder the prediction and generalization capability of a DTI model.
Revised: To summarize, we show that considering DTI as independent entities introduces potential sampling biases which may hinder the prediction and generalization capability of a DTI model.

Original: Correspondingly, we propose a simple approach to gauge the generalization capability of a DTI model emphasizing the evaluation on OOD scenarios.
Revised: Correspondingly, we propose a simple approach to gauge the generalization capability of a DTI model emphasizing the evaluation on OOD scenarios.

Original:  M ETHODS 2. D ATASETS
Revised:  M ETHODS 2. D ATASETS

Original: All the datasets in this work are derived from BindingDB (Liu et al., 2007): a publicly accessible and regularly updated collection of binding afﬁnity values between proteins considered to be drugtargets, and drug-like molecules.
Revised: All the datasets in this work are derived from BindingDB (Liu et al., 2007): a publicly accessible and regularly updated collection of binding afﬁnity values between proteins considered to be drugtargets, and drug-like molecules.

Original: In particular, we adopt two benchmark datasets derived from
Revised: In particular, we adopt two benchmark datasets derived from

Original: BindingDB, one released by Yingkai Gao et al.
Revised: BindingDB, one released by Yingkai Gao et al.

Original: (2018) and the other as deﬁned by Karimi et al.
Revised: (2018) and the other as deﬁned by Karimi et al.

Original: (2019), which have been used for benchmarking recent DTI predictors (Chen et al., 2020;
Revised: (2019), which have been used for benchmarking recent DTI predictors (Chen et al., 2020;

Original: Born et al., 2022).
Revised: Born et al., 2022).

Original: Both benchmark datasets are outlined in Table 1.
Revised: Both benchmark datasets are outlined in Table 1.

Original: The ﬁrst version, which we name BindingDB S , contains 59,136 interactions (involving 48,084 drugs and 798 targets), distributed in training ( train S ), validation ( val S ), test ( test S ) set.
Revised: The ﬁrst version, which we name BindingDB S , contains 59,136 interactions (involving 48,084 drugs and 798 targets), distributed in training ( train S ), validation ( val S ), test ( test S ) set.

Original: It was assembled with the intent of simulating practical scenarios, i.e. “given a pair of drug and target at testing time, the drug, the target, or both of them may have not been observed at training time”
Revised: It was assembled with the intent of simulating practical scenarios, i.e. “given a pair of drug and target at testing time, the drug, the target, or both of them may have not been observed at training time”

Original: (Yingkai Gao et al., 2018).
Revised: (Yingkai Gao et al., 2018).

Original: We remove from the original sets any drug and target exceeding the length constraints of DeepAfﬁnity (Karimi et al., 2019)
Revised: We remove from the original sets any drug and target exceeding the length constraints of DeepAfﬁnity (Karimi et al., 2019)

Original: (see Section 2.2).
Revised: (see Section 2.2).

Original: The second version, “BindingDB L ” , contains 472,925 interactions (involving 321,950 drugs and 3,350 targets), distributed in training ( train L ), validation ( val L ), test ( test L ), and 4 OOD sets, i.e. ion
Revised: The second version, “BindingDB L ” , contains 472,925 interactions (involving 321,950 drugs and 3,350 targets), distributed in training ( train L ), validation ( val L ), test ( test L ), and 4 OOD sets, i.e. ion

Original: Channels ( test Ch ), nuclear Estrogen Receptors ( test ER ), G-Protein-Coupled Receptors ( test GPCR ), and Receptor Tyrosine Kinases ( test RTK ).
Revised: Channels ( test Ch ), nuclear Estrogen Receptors ( test ER ), G-Protein-Coupled Receptors ( test GPCR ), and Receptor Tyrosine Kinases ( test RTK ).

Original: This collection of datasets was released for assessing various Deep Learning techniques and their generalization capability, constraining four protein families only in the respective OOD sets (Karimi et al., 2019), i.e. leaving ion Channels, nuclear Estrogen
Revised: This collection of datasets was released for assessing various deep learning techniques and their generalization capability, constraining four protein families only in the respective OOD sets (Karimi et al., 2019), i.e. leaving ion Channels, nuclear Estrogen

Original: Receptors ( test ER ), G-Protein-Coupled Receptors, and Receptor Tyrosine Kinases out of the training set.
Revised: Receptors ( test ER ), G-Protein-Coupled Receptors, and Receptor Tyrosine Kinases out of the training set.

Original: Although none of the method in this work needs a validation set, we follow the approach used by the authors of DeepAfﬁnity (Karimi et al., 2019), and randomly select 10% of the samples in the training set to split the validation set. 
Revised: Although none of the method in this work needs a validation set, we follow the approach used by the authors of DeepAfﬁnity (Karimi et al., 2019), and randomly select 10% of the samples in the training set to split the validation set. [Table]

Original: 2. DTI P REDICTORS
Revised: 2. DTI P REDICTORS

Original: In this work, we assess the predictive performance of three recent Deep Learning methods, together with a Machine Learning baseline:
Revised: In this work [Table] we assess the predictive performance of three recent deep learning methods, together with a machine learning baseline:

Original: DeepAfﬁnity (Karimi et al., 2019), DeepConv-DTI (Lee et al., [Table] , TransformerCPI (Chen et al., 2020) [Table]
Revised: DeepAfﬁnity (Karimi et al., 2019), DeepConv-DTI (Lee et al., 2019), TransformerCPI (Chen et al., 2020), and Random Forest (Ho, 1995).

Original: and Random Forest (Ho, 1995).

Original: All Deep Learning methods are published along with the code for training and assessing them.
Revised: All deep learning methods are published along with the code for training and assessing them.

Original: We do not perform any hyperparameter optimization and aim to reproduce the methods as described in the respective manuscripts.
Revised: We do not perform any hyperparameter optimization and aim to reproduce the methods as described in the respective manuscripts.

Original: DeepAfﬁnity (Karimi et al., 2019) is the only regressor in this work, which makes comparison to the other methods challenging.
Revised: DeepAfﬁnity (Karimi et al., 2019) is the only regressor in this work, which makes comparison to the other methods challenging.

Original: However, for completeness, we include comparison with this method.
Revised: However, for completeness, we include comparison with this method.

Original: To do this, we impose a classiﬁcation threshold of 0.5 for BindingDB S (which was released already binarized), and of 6 for BindingDB L (as done previously on BindingDB S by Yingkai Gao et al.
Revised: To do this, we impose a classiﬁcation threshold of 0.5 for BindingDB S (which was released already binarized), and of 6 for BindingDB L (as done previously on BindingDB S by Yingkai Gao et al.

Original: DeepAfﬁnity requires the PDB (Berman et al., 2000), Pfam (Mistry et al., 2021) and
Revised: DeepAfﬁnity requires the PDB (Berman et al., 2000), Pfam (Mistry et al., 2021) and

Original: UniRef (Suzek et al., 2015) for building the vocabulary of 72 four-mers describing the targets.
Revised: UniRef (Suzek et al., 2015) for building the vocabulary of 72 four-mers describing the targets.

Original: Drugs are described using an alphabet of 68 letters derived from the corresponding SMILES.
Revised: Drugs are described using an alphabet of 68 letters derived from the corresponding SMILES.

Original: Moreover,
Revised: Moreover,

Original: UniRef50 (Suzek et al., 2015) and Stitch (Szklarczyk et al., 2016) are used for pre-training the embeddings of drugs and targets, respectively.
Revised: UniRef50 (Suzek et al., 2015) and Stitch (Szklarczyk et al., 2016) are used for pre-training the embeddings of drugs and targets, respectively.

Original: DeepAfﬁnity accepts targets of up to 1,500 amino acids, and SMILES of up to 100 symbols.
Revised: DeepAfﬁnity accepts targets of up to 1,500 amino acids, and SMILES of up to 100 symbols.

Original: DeepConv-DTI (Lee et al., 2019) is the only method not relaying on external databases, nor kmers, nor pre-training.
Revised: DeepConv-DTI (Lee et al., 2019) is the only method not relaying on external databases, nor kmers, nor pre-training.

Original: Although it relays on an embedding for representing protein sequences, such embedding is trained from scratch on the training set.
Revised: Although it relays on an embedding for representing protein sequences, such embedding is trained from scratch on the training set.

Original: Drugs are represented by Morgan ﬁngerprints of radius 2 (Rogers & Hahn, 2010).
Revised: Drugs are represented by Morgan ﬁngerprints of radius 2 (Rogers & Hahn, 2010).

Original: It accepts targets of up to 2,500 amino acids.
Revised: It accepts targets of up to 2,500 amino acids.

Original: TransformerCPI (Chen et al., 2020) is inspired from a Transformer architecture (Vaswani et al., 2017), although the drugs are passed in input only to the decoder.
Revised: TransformerCPI (Chen et al., 2020) is inspired from a Transformer architecture (Vaswani et al., 2017), although the drugs are passed in input only to the decoder.

Original: Protein sequences are split into overlapping three-mers of amino acids, and then passed to a word2vec embedding (Mikolov et al., 2013) (pre-trained on human proteins from UniProt (The UniProt Consortium, 2017)).
Revised: Protein sequences are split into overlapping three-mers of amino acids, and then passed to a word2vec embedding (Mikolov et al., 2013) (pre-trained on human proteins from UniProt (The UniProt Consortium, 2017)).

Original: Each atom of the drugs is represented via 34 chemical and physical properties calculated with RDKit (RDKit), which are then passed to a graph. 
Revised: Each atom of the drugs is represented via 34 chemical and physical properties calculated with RDKit (RDKit), which are then passed to a graph. [Figure]

Original: Random Forest (Ho, 1995) is implemented using the code released along with DeepAfﬁnity (Karimi et al., 2019).
Revised: Random Forest (Ho, 1995) is implemented using the code released along with DeepAfﬁnity (Karimi et al., 2019).

Original: In particular, protein sequences are represented by a vector of size 2,500 (padded with zeros for shorter sequences), where each amino acid is encoded via a label encoder, i.e. mapped to a number from one to twenty-ﬁve.
Revised: In particular, protein sequences are represented by a vector of size 2,500 (padded with zeros for shorter sequences), where each amino acid is encoded via a label encoder, i.e. mapped to a number from one to twenty-ﬁve.

Original: Drugs are represented using Morgan ﬁngerprints of radius(Rogers & Hahn, 2010) calculated with RDKit (RDKit).[Figure]
Revised: Drugs are represented using Morgan ﬁngerprints of radius(Rogers & Hahn, 2010) calculated with RDKit (RDKit).

Original: 2. M ETRICS
Revised: 2. M ETRICS

Original: We use Matthews Correlation Coefﬁcient (MCC) to measure predictive performance.
Revised: We use Matthews Correlation Coefﬁcient (MCC) to measure predictive performance.

Original: MCC summarizes the confusion matrix in a single value, and is considered to be more informative than other metrics on imbalanced datasets (Brown, 2018).
Revised: MCC summarizes the confusion matrix in a single value, and is considered to be more informative than other metrics on imbalanced datasets (Brown, 2018).

Original: Speciﬁcally, the MCC can be calculated from the confusion matrix as follow:
Revised: Speciﬁcally, the MCC can be calculated from the confusion matrix as follow:

Original: [Equation]
Revised: [Equation]

Original: with TP (True Positives), TN (True Negatives), FP (False Positives), and FN (False Negatives).
Revised: with TP (True Positives), TN (True Negatives), FP (False Positives), and FN (False Negatives).

Original: The
Revised: The

Original: MCC can assume values from -1 to 1, where the extremes indicate perfect agreement or disagreement between ground truth and model predictions, and 0 indicates no relationship.
Revised: MCC can assume values from -1 to 1, where the extremes indicate perfect agreement or disagreement between ground truth and model predictions, and 0 indicates no relationship.

Original: For completeness, we include Accuracy and F1 metrics for all results in this work in the AppendixA.
Revised: For completeness, we include Accuracy and F1 metrics for all results in this work in the AppendixA.

Original:  R ESULTS 3. DTI BENCHMARK
Revised:  R ESULTS 3. DTI BENCHMARK

Original: We benchmark DeepAfﬁnity, DeepConv-DTI, TransformerCPI, and a Random Forest on
Revised: We benchmark DeepAfﬁnity, DeepConv-DTI, TransformerCPI, and a Random Forest on

Original: BindingDB S and BindingDB L .
Revised: BindingDB S and BindingDB L .

Original: To do so, we train all methods twice to obtain a model for each benchmark dataset.
Revised: To do so, we train all methods twice to obtain a model for each benchmark dataset.

Original: We observe good performance across the board on test S and test L , as shown in Figure 1a.
Revised: We observe good performance across the board on test S and test L , as shown in Figure 1a.

Original: DeepConv-DTI achieves a MCC over 0.7, slightly outperforming other methods.
Revised: DeepConv-DTI achieves a MCC over 0.7, slightly outperforming other methods.

Original: Notably, the Random Forest is never the worst method in this setting, and appears to be on par with the Deep Learning methods.
Revised: Notably, the Random Forest is never the worst method in this setting, and appears to be on par with the deep learning methods.

Original: Lower performance is observed regarding generalization, as measured on the four OOD sets, i.e. test Ch , test ER , test GPCR , and test RTK (see Figure 1b).
Revised: Lower performance is observed regarding generalization, as measured on the four OOD sets, i.e. test Ch , test ER , test GPCR , and test RTK (see Figure 1b).

Original: All methods achieve lower MCC values than on test S and test L , and exceed a MCC value of 0.3 only in two cases, i.e. DeepAfﬁnity on test Ch , and DeepConv-DTI on test RTK .
Revised: All methods achieve lower MCC values than on test S and test L , and exceed a MCC value of 0.3 only in two cases, i.e. DeepAfﬁnity on test Ch , and DeepConv-DTI on test RTK .

Original: DeepConv-DTI achieves the highest MCC values in most cases, although no method outperforms the competitors on all four OOD sets, and the Random
Revised: DeepConv-DTI achieves the highest MCC values in most cases, although no method outperforms the competitors on all four OOD sets, and the Random

Original: Forest is generally the worst by MCC value.
Revised: Forest is generally the worst by MCC value.

Original: The results of our benchmark outline that no method outperforms all the other methods assessed, in all benchmarking scenarios.
Revised: The results of our benchmark outline that no method outperforms all the other methods assessed, in all benchmarking scenarios.

Original: Importantly, all methods achieve good MCC on test S and test L , but much lower performance on OOD scenarios, pointing towards poor generalization.
Revised: Importantly, all methods achieve good MCC on test S and test L , but much lower performance on OOD scenarios, pointing towards poor generalization.

Original: Regarding
Revised: Regarding

Original: DeepAfﬁnity, we observe particularly good predictive performance on test L and test Ch , although it is challenging to compare the only regressor with the other methods.[Figure]
Revised: DeepAfﬁnity, we observe particularly good predictive performance on test L and test Ch , although it is challenging to compare the only regressor with the other methods.

Original: 3. D RUGS AND T ARGETS DISTRIBUTION
Revised: 3. D RUGS AND T ARGETS DISTRIBUTION

Original: To identify possible sources of overoptimistic performance on test S and test L , which may justify the poor generalization on test Ch , test ER , test GPCR , and test RTK , we look deeper in the composition of the benchmark datasets.
Revised: To identify possible sources of overoptimistic performance on test S and test L , which may justify the poor generalization on test Ch , test ER , test GPCR , and test RTK , we look deeper in the composition of the benchmark datasets.

Original: In Section 3.2.1, we explore the overlap between training and testing sets, ﬁrst at the level of interactions and then at the level of their constitutive drugs and targets.
Revised: In Section 3.2.1, we explore the overlap between training and testing sets, ﬁrst at the level of interactions and then at the level of their constitutive drugs and targets.

Original: In Section 3.2.2, we performed a baseline analysis to measure the potential information leakage because of the overlap of drugs and targets across different interactions between training and testing sets.
Revised: In Section 3.2.2, we performed a baseline analysis to measure the potential information leakage because of the overlap of drugs and targets across different interactions between training and testing sets.

Original: Our results show that treating DTIs as independent entities causes information leakage in the context of train-test splits.
Revised: Our results show that treating DTI as independent entities causes information leakage in the context of train-test splits.

Original: 3.2. O VERLAPS ACROSS SETS
Revised: 3.2. O VERLAPS ACROSS SETS

Original: We ﬁnd overlapping interactions in BindingDB L , a simple case of data leakage.
Revised: We ﬁnd overlapping interactions in BindingDB L , a simple case of data leakage.

Original: Speciﬁcally, we observe that 4% of the interactions in both test L and val L are present in train L (see Figure 4a).
Revised: Speciﬁcally, we observe that 4% of the interactions in both test L and val L are present in train L (see Table 1).

Original: We also observe DTIs with multiple labels, i.e. the same drug-target pair associated to multiple binding observations.
Revised: We also observe DTI with multiple labels, i.e. the same drug-target pair associated to multiple binding observations.

Original: This case of data leakage may be due to the lack of stereochemistry in the SMILES of
Revised: This case of data leakage may be due to the lack of stereochemistry in the SMILES of

Original: BindingDB L , leading different molecules to be treated as if they were the same.
Revised: BindingDB L , leading different molecules to be treated as if they were the same.

Original: Beyond direct overlap of interactions between datasets, we also look at interactions not as independent entities, but in terms of their constitutive drugs and targets (see Table 1).
Revised: Beyond direct overlap of interactions between datasets, we also look at interactions not as independent entities, but in terms of their constitutive drugs and targets (see Table 1).

Original: In this regard, we ﬁnd overlap of drugs and targets across all datasets with respect to their training set, including test Ch , test ER , test GPCR , and test RTK .
Revised: In this regard, we ﬁnd overlap of drugs and targets across all datasets with respect to their training set, including test Ch , test ER , test GPCR , and test RTK .

Original: We ﬁnd that 30% of drugs and 90% of targets in test S are also in train S , as shown in Figure 2a.
Revised: We ﬁnd that 30% of drugs and 90% of targets in test S are also in train S , as shown in Table 1.

Original: We ﬁnd an even greater overlap between test L and trarin L , i.e. 37% of drugs and 95% of targets, as shown in Figure 2b.
Revised: We ﬁnd an even greater overlap between test L and train L , i.e. 37% of drugs and 95% of targets.

Original: Similar overlaps exist also between the validation sets and the respective training sets for both BindingDB S and BindingDB L .
Revised: Similar overlaps exist also between the validation sets and the respective training sets for both BindingDB S and BindingDB L , as shown in Figure 2.

Original: In the case of the four OOD sets, we ﬁnda that 13-25% of the drugs in these sets, as well as one target in test Ch and a second one in test RTK , are present in train L .
Revised: In the case of the four OOD sets, we ﬁnd that 13-25% of the drugs in these sets, as well as one target in test Ch and a second one in test RTK , are present in train L .

Original: We further investigate the large overlaps of targets across train, validation, and test set seen in
Revised: We further investigate the large overlaps of targets across training, validation, and test set seen in

Original: BindingDB S and BindingDB L , looking at the most represented targets.
Revised: BindingDB S and BindingDB L , looking at the most represented targets.

Original: In particular, we pick the top 10% targets by number of interactions in each dataset, i.e. considering the number of interactions in each dataset individually.
Revised: In particular, we pick the top 10% targets by number of interactions in each dataset, i.e. considering the number of interactions in each dataset individually.

Original: We observe that nearly all top targets in test S and test L are among the top targets in the respective training and validation set, as shown in Figure 3.
Revised: We observe that nearly all top targets in test S and test L are among the top targets in the respective training and validation set, as shown in Figure 3.

Original: Thus, the overlap of targets across train, validation, and test set seen in BindingDB S and BindingDB L is strongly present even when looking at the top targets.[Figure]
Revised: Thus, the overlap of targets across training, validation, and test set seen in BindingDB S and BindingDB L is strongly present even when looking at the top targets.

Original: 3.2. I DENTITY MATCHING
Revised: 3.2. I DENTITY MATCHING

Original: The overlaps of drugs and targets we observe across the benchmark datasets may represent a problematic case of information leakage, which may hinder the learning of DTI model.
Revised: The overlaps of drugs and targets we observe across the benchmark datasets may represent a problematic case of information leakage, which may hinder the learning of DTI model.

Original: We investigate this hypothesis performing a baseline analysis by extracting the average activity value for each drug and target in the training set.
Revised: We investigate this hypothesis performing a baseline analysis by extracting the average activity value for each drug and target in the training set.

Original: Then, for each interaction in the test set, if it contains a drug or target that is present in the training set, a drug-based prediction (p drug) and/or a targetbased prediction (p target) is provided, corresponding to their average activity in the training set.
Revised: Then, for each interaction in the test set, if it contains a drug or target that is present in the training set, a drug-based prediction (p drug) and/or a targetbased prediction (p target) is provided, corresponding to their average activity in the training set.

Original: This baseline analysis matches identities for drugs and targets across training and test sets; it performs no Machine Learning, does not access any protein sequence or compound chemical information, nor does it use any similarity or clustering calculation.
Revised: This baseline analysis matches identities for drugs and targets across training and test sets; it performs no machine learning, does not access any protein sequence or compound chemical information, nor does it use any similarity or clustering calculation.

Original: Thus, this baseline cannot analyze interactions involving a drug and a target when both are not present in the training set (see Tablefor the number of predictions for each dataset).
Revised: Thus, this baseline cannot analyze interactions involving a drug and a target when both are not present in the training set (see Table [Figure] [Figure] [Figure]

Revised: for the number of predictions for each dataset).

Original: Given the lack of chemical and protein sequence information in this approach, one would expect a performance close to random (MCC close to 0.0).
Revised: Given the lack of chemical and protein sequence information in this approach, one would expect a performance close to random (MCC close to 0.0).

Original: Nonetheless, this approach was able to produce surprisingly high MCC values on both BindingDB S and BindingDB L , as shown in Figure 4b (see full results in Table 6).
Revised: Nonetheless, this approach was able to produce surprisingly high MCC values on both BindingDB S and BindingDB L , as shown in Figure 4b (see full results in Table 6).

Original: Therefore, this analysis provides evidence of the information leakage present in the dataset due to the non-independent nature of interactions with respect to their constitutive drugs and targets.
Revised: Therefore, this analysis provides evidence of the information leakage present in the dataset due to the non-independent nature of interactions with respect to their constitutive drugs and targets.

Original: 3. E VALUATION OF G ENERALIZATION C APABILITY
Revised: 3. E VALUATION OF G ENERALIZATION C APABILITY

Original: Motivated by the information leakage we ﬁnd in BindingDB S and
Revised: Motivated by the information leakage we ﬁnd in BindingDB S and

Original: BindingDB L , we investigate whether ﬁltering the test samples according to the overlap with the training set provides a more informative evaluation of the generalization capability of a method.
Revised: BindingDB L , we investigate whether ﬁltering the test samples according to the overlap with the training set provides a more informative evaluation of the generalization capability of a method.

Original: Therefore, we propose to disaggregate OOD scenarios to improve the benchmarking of DTI predictors.
Revised: Therefore, we propose to disaggregate OOD scenarios to improve the benchmarking of DTI predictors.

Original: In particular, we derive 3 subsets for each test set collecting any interaction where:
Revised: In particular, we derive 3 subsets for each test set collecting any interaction where:

Original: the drug is not in the training set ( D ); 2.
Revised: the drug is not in the training set ( D ); 2.

Original: the target is not in the training set ( T ); 3.
Revised: the target is not in the training set ( T ); 3.

Original: neither the target nor the drug is in the training set ( I ).
Revised: neither the target nor the drug is in the training set ( I ).

Original: In the Sections below, we use this approach to shed additional light on the information leakage that partially causes the overoptimistic predictive performance observed on test S and test L .
Revised: In the Sections below, we use this approach to shed additional light on the information leakage that partially causes the overoptimistic predictive performance observed on test S and test L .

Original: 3.3. B INDING DB S
Revised: 3.3. B INDING DB S

Original: The ﬁltering approach we propose results in three subsets of test S , i.e. D S , T S , and I S (see Table 2).
Revised: The ﬁltering approach we propose results in three subsets of test S , i.e. D S , T S , and I S (see Table 2).

Original: For all methods, we observe good predictive performance on test S and D S , as shown in Figure 5a.[Figure]
Revised: For all methods, we observe good predictive performance on test S and D S , as shown in Figure 5a.

Original: [Table] Lower performance is shown on T S and I S , meaning that ﬁltering by target has a similar effect to ﬁltering by both drug and target in this dataset.
Revised: Lower performance is shown on T S and I S , meaning that ﬁltering by target has a similar effect to ﬁltering by both drug and target in this dataset.

Original: The stronger relevance of ﬁltering by target matches the larger overlap seen across targets for D S (as show in Table 2).
Revised: The stronger relevance of ﬁltering by target matches the larger overlap seen across targets for D S (as show in Table 2).

Original: In part, this can be expected due to the composition of the data (one order of magnitude fewer targets than drugs), but it also points towards the information leakage present in test S , due to the non-independent nature of drugs and targets within interactions. 
Revised: In part, this can be expected due to the composition of the data (one order of magnitude fewer targets than drugs), but it also points towards the information leakage present in test S , due to the non-independent nature of drugs and targets within interactions. [Table]

Original: [Figure]
Revised: [Figure]

Original: 3.3. B INDING DB L
Revised: 3.3. B INDING DB L

Original: The large overlap between train L and test L leaves only 64 interactions in I L .
Revised: The large overlap between train L and test L leaves only 64 interactions in I L .

Original: Therefore, we create a new benchnmark dataset, called BindingDB R , by resplitting the training, validation, and test set of BindingDB L (see Table 2).
Revised: Therefore, we create a new benchnmark dataset, called BindingDB R , by resplitting the training, validation, and test set of BindingDB L (see Table 2).

Original: To do so, we randomly select 10% of targets and 10% of drugs, and allocate any interaction involving those to test R .
Revised: To do so, we randomly select 10% of targets and 10% of drugs, and allocate any interaction involving those to test R .

Original: We allocate 90% of any other interaction to train R , and the remaining 10% to val R .
Revised: We allocate 90% of any other interaction to train R , and the remaining 10% to val R .

Original: Thus, all samples in test R are in D R , T R , or I R .
Revised: Thus, all samples in test R are in D R , T R , or I R .

Original: BindingDB R contains a similar ratio of samples as in BindingDB L , while reducing the overlap of both drugs and targets between training and test set, and remediating the data leakage in test L (see
Revised: BindingDB R contains a similar ratio of samples as in BindingDB L , while reducing the overlap of both drugs and targets between training and test set, and remediating the data leakage in test L (see

Original: Figure 4a).
Revised: Figure 4a).

Original: We also ﬁlter the four OOD sets removing any interaction involving a drug or target in train R , i.e. I
Revised: We also ﬁlter the four OOD sets removing any interaction involving a drug or target in train R , i.e. I

Original: Ch , I ER , I GPCR , and I RTK .
Revised: Ch , I ER , I GPCR , and I RTK .

Original: We train a Random Forest on train R and we expect analogous results for the Deep Learning methods in this study.
Revised: We train a Random Forest on train R and we expect analogous results for the deep learning methods in this study.

Original: The Random Forest performs similarly well on test R and D R , and worse on T R and
Revised: The Random Forest performs similarly well on test R and D R , and worse on T R and

Original: I R , as shown in Figure 5b.
Revised: I R , as shown in Figure 5b.

Original: This matches what we observe for BindingDB S in Section 3.3.1, and provides additional evidence of the non-independence of drugs and targets within interactions.
Revised: This matches what we observe for BindingDB S in Section 3.3.1, and provides additional evidence of the non-independence of drugs and targets within interactions.

Original: The predictive performance on T R and I R , and to same extend those on T S and I S , resemble the results on test Ch , test ER , test GPCR , and test RTK .
Revised: The predictive performance on T R and I R , and to same extend those on T S and I S , resemble the results on test Ch , test ER , test GPCR , and test RTK .

Original: Thus, we see our approach as an alternative solution to evaluate OOD predictive performance, without depriving the training set of entire protein families (as done for the four OOD sets).
Revised: Thus, we see our approach as an alternative solution to evaluate OOD predictive performance, without depriving the training set of entire protein families (as done for the four OOD sets).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we perform an independent benchmark of recent DTI predictors based on Deep Learning.
Revised: In this work, we perform an independent benchmark of recent DTI predictors based on deep learning.

Original: We ﬁnd information leakage in previously used DTI benchmark datasets due to the nonindependent nature of drugs and targets within interactions.
Revised: We ﬁnd information leakage in previously used DTI benchmark datasets due to the non-independent nature of drugs and targets within interactions.

Original: We examine if such information leakage is related to overoptimistic predictive performance and relatively poor generalization observed.
Revised: We examine if such information leakage is related to overoptimistic predictive performance and relatively poor generalization observed.

Original: To support this idea, we show the high predictive performance obtained with a baseline analysis based solely on drugs and targets identity.
Revised: To support this idea, we show the high predictive performance obtained with a baseline analysis based solely on drugs and targets identity.

Original: To remediate this source of bias, we propose a novel approach for assessing DTI predictors with respect to the content of their training set.
Revised: To remove this source of bias and improve the assessment of DTI predictors, we propose a novel approach for assessing DTI predictors with respect to the content of their training set.

Revised: Our novel approach results in a more comprehensive assessment of DTI predictors, discerning the OOD scenarios seen at test time, and highlighting potential source of bias without altering the training set.

Original: For future work, we aim to consider similarity of drugs and targets, instead of identity, to extend our minimal solution for evaluating the generalization capability of DTI models.
Revised: For future work, we aim to consider similarity of drugs and targets, instead of identity, to extend our minimal solution for evaluating the generalization capability of DTI predictors.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: The authors are grateful to Brian Kidd and Wilbert Copeland  for useful comments and suggestions.
Revised: The authors are grateful to Brian Kidd and Wilbert Copeland from BMS, and the anonymous reviewers for useful comments and suggestions.