Original: Glolloc: Mixture of Global and Local Experts for Molecular Activity Prediction
Revised: G LOLLOC : M IXTURE OF G LOBAL AND L OCAL E XPERTS FOR M OLECULAR A CTIVITY P REDICTION

Original: Anonymous authors Paper under double-blind review

Original: Abstract
Revised: A BSTRACT

Original: Quantitative structure-activity relationships (QSAR) models have been used for decades to predict the activity of small molecules, using encodings of the molecular structure, for which simple 2D descriptors of the molecular graph are still most commonly used.
Revised: Quantitative structure-activity relationships (QSAR) models have been used for decades to predict the activity of small molecules, using encodings of the molecular structure, for which simple 2D descriptors of the molecular graph are still most commonly used.

Original: One of the recurrent problems of QSARmodels is that relationships observed for a specific scaffold (pruned molecular skeleton) are sometimes not translatable to another, due to the 3D flexibility of molecular objects.
Revised: One of the recurrent problems of QSAR is that relationships observed for a specific scaffold (pruned molecular skeleton) are often not transferable to another; this is often addressed by building several local models from subsets of the chemical space.

Original: This is also true when building multitask networks predicting the activity  against several proteins at the same time - sometimes single protein models work better, and adding dissimilar proteins into the model decreases performance.
Revised: Similarly, single task models sometimes outperform large multi-task models in predicting the activity of small molecules against specific proteins.

Original: In this paper, mixtures of experts (MoE) are used to combine a global network and local structures of the dataset (e.g. molecular scaffold, single protein) in the single task or multitask framework.
Revised: In this paper, we introduce Glolloc, a global-local MoE-QSAR architecture, based on a Mixture of Experts (MoE) framework.

Original: We show that structuring the learning process with protein or chemical series information can enhance model performance and provide a built-in model introspection tool.
Revised: Glolloc combines predictions from global and local experts, provides a built-in model introspection tool, can enhance model performance, and removes the need to maintain several local models.

Original:  Introduction
Revised:  I NTRODUCTION

Original: Mixtures of experts (MoE) are a subset of ensemble learning algorithms, consisting in individual learners or experts, and a gate that assigns weights to different learners.
Revised: Mixtures of experts (MoE) are a subset of ensemble learning algorithms, consisting in individual learners or experts, and a gate that assigns weights to different learners.

Original: They were relatively popular in the 90s (1; 2), and have recently regained interest in the deep learning community, notably for language and vision (3; 4).
Revised: They were relatively popular in the 90s (Jordan & Jacobs, 1993; Jacobs et al., 1991), and have recently regained interest in the deep learning community, notably for language and vision (Ruiz et al., 2021; Shazeer et al., 2017).

Original: MoE can be seen as a type of meta-learning approach as it "learns to learn" from an ensemble of experts, which can be explicitly pre-defined e.g. by using clustering or subsets of input features (5; 6).
Revised: MoE can be seen as a type of meta-learning approach as it ”learns to learn” from an ensemble of experts, which can be explicitly pre-defined e.g. by using clustering or subsets of input features (Gutta et al., 2000; Tang et al., 2002).

Original: Inthe field of quantitative structure-activity relationships (QSAR), where the aim is to build predictive models from chemical structures (e g. activityof molecule against a protein of interest), local models trained on subsets of molecules often show better performances than global modelsthat are trained with the whole dataset (7).
Revised: In quantitative structure-activity relationships (QSAR) models, where the aim is to predict activity from molecular structures (e.g. activity against a protein of interest), local models trained on subsets of molecules often provide better performances than global models trained on the whole dataset (Yuan et al., 2006).

Original: Ina drug discovery project focused models are often constructed for several chemical series (sets of molecules with a similar skeleton or "scaffold").
Revised: In drug discovery projects, a local model is often constructed for each molecular series of interest (set of molecules with a similar skeleton or ”scaffold”).

Original: However, throwing away most of the data comes with a loss of potentially useful information.
Revised: At a given stage, several chemical series might be investigated, and several models are separately optimised and maintained. However, these local models discard potentially valuable information, and do not always perform better than global models. Furthermore, some molecules share characteristics from different series and would benefit from combining several local model predictions.

Original: Methods that are able to combine advantages from  local and global models should therefore be of particular interest in the field.
Revised: Methods that are able to combine advantages from both local and global models should therefore be of particular interest in the field.

Original: Another potential application for problem partitioning would be multitask learning for ligand-protein interactions or proteochemometric modelling - i.e. predicting the activity of a small molecule against several proteins with the same model.
Revised: We thought a MoE architecture could be a way to combine predictions from global and local experts, each local expert being specialised in a chemical series of interest. Local experts for QSAR can also be defined in other ways. When the goal is to predict the activity of a small molecule against several proteins at the same time, one local expert can be assigned to each protein task.

Original: These multi-protein models can perform better or worse than single task learning, depending on the protein (8) - subdividing the problem and training one expert per protein or protein family, in addition to a global expert trained on everything, could help to better adapt predictions to specific proteins.
Revised: Multi-protein models can perform better or worse than single task learning, depending on the protein (Gaspar et al., 2021) - subdividing the problem and training one expert per protein or protein family, in addition to a global expert trained on everything, could help to better adapt predictions to specific proteins.

Original: The MoE literature for quantitative structure-activity relationships (QSAR) is relatively scarce - MoEs were recently investigated (9) as a QSAR ensembling method across different feature spaces.
Revised: The MoE literature for QSAR is very scarce - MoEs were recently investigated (D¨org˝o et al., 2020) as a QSAR ensemble method across different feature spaces.

Original: In methods introduced in the following sections, the problem is naively partitioned based on input instances rather than input features, although combining both approaches would be an interesting follow-up.
Revised: In methods introduced in the following sections, the problem is naively partitioned by input instances rather than input features, although combining both approaches would be an interesting follow-up.

Original: Attempting to pre-partition the input space based on chemistry expert knowledge and then using a MoE architecture to choose between local and global experts has not yet been attempted.
Revised: Partitioning the input space based on chemistry expert knowledge, and then using a MoE architecture to choose between local and global experts has not yet been attempted.

Original: The main aim of the following experiments is to explore differentautomated ways to partition the problem in away that is chemically relevant, and come up with a model capable to focus on a chemical series or protein of interest whilst not discarding valuable information
Revised: The main aim of the following experiments is to explore different ways to partition the problem in a chemically relevant way, and come up with a model able to focus on local aspects whilst not discarding valuable information.

Original: [Table]  Methods 2. Data and benchmark goals [Table] [Table]
Revised: The main contribution of this paper is global-local (”Glolloc”) MoE-QSAR architecture for single or multi-task learning that: (a) combines advantages from both local and global models, which are routinely used separately in QSAR pipelines, (b) is structured using expert knowledge (e.g. medicinal chemist partitioning the space by chemical series), (c) removes the need to maintain many local models, (d) determines which experts are more appropriate for a given molecule, (e) provides a built-in model introspection tool via weights given to experts, and (f) can improve performance over a single global model or an ensemble of random experts.

Original:  In the following experiments
Revised:  M ETHODS 2. D ATA

Original: [Table]
Revised: [Table]

Original:  the 7 datasets (Table 1) correspond to 7 diverse proteins from the diverse subset of the DUD-E data collection (11). Note that while the diverse subset from DUD-E was used to choose the proteins for the targets, the ligand dataassociated with those proteins is not suitable for a topological QSAR experiment. Consequently, binding data was retrievedfor each protein from a publicly available benchmark dataset (12) curated by 
Revised: In the single task experiments, the 7 datasets (Table 1) correspond to 7 diverse proteins from the diverse subset of the DUD-E data collection (Mysinger et al., 2012). Note that whilst the diverse subset from DUD-E was used to choose the proteins, the ligand data was retrieved from a publicly available benchmark dataset (Lenselink et al., 2017b) curated by Lenselink et al.

Original: Lenselink et al. (13), which compiles activity of small molecules extracted from the ChEMBL database (14).
Revised: (Lenselink et al., 2017a), which compiles activities of small molecules extracted from the ChEMBL database (Gaulton et al., 2016).

Original: Instances are small molecules, and the endpoint is pChEMBL activity (-Log(molar IC50,
Revised: For our purposes, instances are small molecules, and the endpoint is pChEMBL activity (-

Original:  XC50, EC50, AC50, Ki, Kd or Potency)) against each of the 7 proteins.
Revised: Log(molar IC50, XC50, EC50, AC50, Ki, Kd or Potency)).

Original: For each of these datasets, a tree-based "autosmarts" algorithm (Algorithm 1) was used on molecules with pChEMBL value > 5.5 to define a top chemical series of interest - its corresponding SMARTS pattern visualisation is reported in Table 1.
Revised: For each of the 7 datasets, a tree-based ”autosmarts” algorithm (Algorithm 1) was used on molecules with pChEMBL value > 5.5 to find a top chemical series of interest - its corresponding SMARTS pattern visualisation is reported in Table 1.

Original: Compounds matching that series were split into train, valid and test using time splits, and the series-specific train set was merged with  the remaining compounds, resulting in a  set that contains bothdiverse molecules and the pattern of interest.
Revised: Compounds matching that pattern were split into train, valid and test using time splits. This train set was augmented with data not matching the pattern, resulting in a train set that contains both the local data and other diverse molecules.

Original: The time splits were defined using publication year, to mimic what happens in a drug discovery project, where the aim is to do better on more recent data.
Revised: The time splits were defined using publication year, to mimic what happens in a drug discovery project, where the aim is to do better on more recent data.

Original: It should be noted that publication year is not an ideal proxy and does not reflect the actual experiment date.
Revised: It should be noted that publication year is not an ideal proxy and does not reflect the actual experiment date.

Original: It is, however, the only publicly available proxy.

Original: The cp3a4 dataset was supplemented by other cytochrome data for the multi-task challenge (CP2D6, CP1A2, CP2C9, and CP2CJ) to construct the cyp dataset - the test dataset being the same as for cp3a4.
Revised: The cp3a4 dataset was supplemented by other cytochrome data for the multi-task challenge (CP2D6, CP1A2, CP2C9, and CP2CJ) to construct the cyp dataset - the test dataset being the same as for cp3a4.

Original: ECFP4_2048 descriptors were used to featurise molecules: Morgan fingerprints with a radius of 2 and size 2048, as implemented in rdkit (15).
Revised: ECFP4 2048 descriptors were used to featurise molecules: Morgan fingerprints with a radius ofand size 2048.

Original: Glolloc architectures (cf.

Original: next sections) were benchmarked against a single global expert baseline and the scikit-learn (16) implementation of Random Forests (17) regression.

Original: Parameters were optimised on the valid set without cross-validation - hidden layer sizes were the only variable hyperparameter for Glolloc networks (two hidden layers with [64, 64] or [1024, 1024] units).

Original: For Random Forests, combinations of hyperparameters were systematically screened from maximum depth = {5, 25, 50, 75, 100} and number of estimators = {100, 200}.

Original: 2. Proposed single task and multi-task networks
Revised: 2. P ROPOSED SINGLE TASK AND MULTI - TASK NETWORKS

Original: Two architectures were investigated: a single-task network with local experts specialising in different regions of the chemical space (Figure 1), and a multi-task network to predict the activity of multiple proteins, where local experts specialise in predicting specific proteins (Figure 2).
Revised: Two architectures were investigated: a single-task network with local experts specialising in different regions of the chemical space (Figure 1), and a multi-task network to predict the activity of multiple proteins, where local experts specialise in predicting specific proteins (Figure 2).

Original: In each case, in addition to the local networks (chemical series or protein experts), a global expert is trained on the whole data set, and is expected to perform better for molecules outside of the applicability domain of the local models.
Revised: In each case, in addition to the local networks (chemical series or protein experts), a global expert is trained on the whole data set, and is expected to perform better for molecules outside of the applicability domain of the local models.

Original: Only fully connected layers were used, with fixed parameters for dropout (0.25 for experts, 0.025 for the gate), early stopping patience (15), batch size (32), learning rate (0.0005), and temperature (1) for the gate softmax - for the datasets in this paper, which are relatively small, the networks are easily trainable on a single CPU within a few minutes. The gate is a fully connected network with a softmax function, outputing a gate weight matrix of dimension [batch size, number of tasks, number of experts], the softmax function being applied on the last dimension, so that weightsacross experts sum to1 for each task.
Revised: The gate network, trained at the same time as the experts, assigns weights to each of them.

Original: The final predicted activity for a molecule n and task t is the weighted sum of predictions from different experts  and gate weights  (Equation 1).
Revised: The final predicted activity for a molecule n and task t is the weighted sum of outputs from different experts ( O ) and gate weights ( G ) (Equation 1).

Original: [Equation]
Revised: [Equation]

Original: 2. Training procedure

Original: Experts are defined using masks, so that the losses for irrelevant data points are not taken into account during the training process: e.g. if an expert focuses on pyrazole-containing molecular compounds, all losses for molecules not containing a pyrazole substructure will be zeroed out.

Original: Experts and gate are trained at the same time, although each expert is kept independent by using expert-specific optimisers and losses (Equation 2).

Original: The gate is also assigned its own optimiser, but is made "aware" of expert predictions in the gate-specific loss (Equation 3), which is used to update the gate parameters.

Original: This expert independence makes training the network challenging, as validation sets need to be carefully defined and loss curves monitored for each expert - local experts are in effect trained on subsets of the training data, and they are prone to overfitting.

Original: This is only partially addressed by adding learning rate decay for each optimiser.

Original: [Equation]
Revised: [Equation]

Original: 2. Automated methods to define chemical series
Revised: 2. A UTOMATED METHODS TO DEFINE CHEMICAL SERIES

Original: In a drug discovery project, chemical series are typically identified by medicinal chemists as structural patterns.
Revised: In a drug discovery project, chemical series are typically identified by medicinal chemists.

Original: Chemoinformaticians convert the patterns into the appropriate SMARTSdepictions -SMARTS being a type of regex language for chemical structures which allow to query molecules containing specific substructures.
Revised: Chemoinformaticians encode them as SMARTS - a type of regex language, which allows to query molecules containing specific substructures.

Original: However, for benchmarks on public datasets, an automated way to partition the chemical space is needed.
Revised: However, for benchmarks on public data sets, an automated way to partition the chemical space is needed.

Original: Note that in a real-world scenario, project chemists would help in the definition of the series to add their domain knowledge of the model structure.
Revised: Note that in a real-world scenario, project chemists would help in the definition of the series to add their domain knowledge to the model structure.

Original: Glolloc implements 3 methods to generate chemical series experts from a chemical dataset: clusterbased, tree-based, and neighbourhood-based.
Revised: Glolloc implements 3 methods to generate chemical series experts from a chemical dataset: cluster-based, tree-based, and neighbourhoodbased.

Original: The cluster-based method is self-explanatory: a k-means clusterer using scikit-learn’s implementation (16) is trained on the training set and cluster labels are used to construct cluster-specific experts.
Revised: The cluster-based method is self-explanatory: a k-means clusterer using scikit-learn’s implementation (Pedregosa et al., 2011) is trained on the training set, and cluster labels are used to construct cluster-specific experts.

Original: The tree-based and neighbourhood-based methods both rely on finding fuzzy maximum substructures (MCS) in groups of molecules, defined by their hierarchical clustering or neighbourhood membership, respectively, and using the RDKit (15) implementation of fuzzy
Revised: The tree-based and neighbourhood-based methods both rely on finding fuzzy maximum substructures (MCS) in groups of molecules.

Original: MCS - with a minimum of 20 molecules per MCS. The advantage of MCS-based methods is that we obtain SMARTS patterns that can be visualised and interpreted easily - this type of approach is coined "autosmarts" in this paper. The tree-based method (Algorithm 1) is similar to the one describedrecently by Kruger al (18): building fuzzy maximum common substructures (MCS) from nodes in a tree obtained using an agglomerative clustering algorithm, going from the root to the leaves: when a series (MCS) is found ina parent node, children are discarded. The difference between the Glolloc implementation and the one by Kruger et al. is the criterion to validate the MCS: they use a probability that a random molecule matches the scaffold, whereas Glolloc sets a minimum number of atoms for the scaffold, to guarantee a large enough scaffold (default value:atoms).
Revised: They are described in further detail in the Appendix.

Original: Strict MCS ring matching rules and loose rules for atom and bond matching were arbitrarily used- the impact of different MCS algorithm settings is not explored in this paper. Algorithm 1 Tree-based autosmarts algorithm: find fuzzy maximum common substructures (MCS) in different regions of the chemical space using hierarchical clustering min _ atoms ← user parameter ▷ minimum number of atoms in MCS min _ node _ size ← user parameter ▷ minimum number of molecules to construct an MCS from T ← dendrogram ( D ) ▷ construct dendrogram from the molecule data matrix D
Revised: We used the tree-based method in this paper to define our chemical series, as we obtained similar results with the neighbourhood-based method.

Original: N ← node _ dict ( T ) ▷ dictionary mapping tree nodes to molecules N ′ ← sort ( N ) ▷ sort nodes from root to leaves MCS ← ∅ ▷ the MCS set stores the maximum common substructures for node, molecules _ in _ node in N ′ do if count ( molecules _ in _ node ) ≥ min _ node _ size then mcs ← fuzzy _ maximum _ substructure ( molecules _ in _ node )
Revised: The advantage of MCS-based methods is the automatically generated SMARTS patterns that can be visualised and interpreted easily - this type of approach is therefore coined ”autosmarts” in this paper.

Original: ▷ if a large enough MCS is found in parent node if mcs exists and mcs.n _ atoms ≥ min _ atoms then MCS ← MCS ∪ { mcs } set children of node in N ′ to ∅ ▷ discard children nodes end if end if end for The neighbourhood-based method implemented in Glolloc (Algorithm 2) is described in the Appendix, as it was not used for benchmarks in this paper.
Revised: [Table]

Original:  Results 3. Single task: partition problem by chemical series
Revised:  R ESULTS 3. S INGLE TASK : PARTITION PROBLEM BY CHEMICAL SERIES

Original: This section compares Random Forests and Glolloc architectures using different partitions of the chemical space to define experts. [Table]
Revised: This section compares Random Forests and Glolloc architectures using different partitions of the chemical space to define experts (Figure 1).

Original: Two control experiments were designed for the Glolloc framework: a model using a single global expert, and a model using one global expert and 6 random experts.
Revised: Two control experiments were designed for the Glolloc framework: a model using a single global expert, and a model using one global expert and 6 random experts.

Original: The random experts divide the chemical space into 6 random sections.
Revised: The random experts divide the chemical space into 6 random sections.

Original: The control experiments are run against two structured Glolloc models: one with the global expert and 6 cluster experts, the other with the global expert and6 autosmarts (Algorithm 1) experts.
Revised: The control experiments are run against two structured Glolloc models: one with the global expert and 6 cluster experts, the other with the global expert and autosmarts (Algorithm 1) experts.

Original: The more structured experts (based on clustering or autosmarts algorithms) yield lower mean squared errors (MSE) for 5 out of 7 datasets, and higher Spearman rank correlation in 6 out of 7 (Table 2).
Revised: The more structured experts (based on clustering or autosmarts algorithms) yield lower mean squared errors (MSE) for 5 out of 7 datasets, and higher Spearman rank correlation in 6 out of 7 (Table 2).

Original: When comparing the best glolloc expert combination (whether random or structured) against a single expert or Random Forests (Figure 3), Random Forests wins only for the gcr dataset and has the worst performance in all other cases.
Revised: When comparing the best glolloc expert combination (whether random or structured) against a single expert or Random Forests (Figure 3), Random Forests wins only for the gcr dataset and has the worst performance in all other cases.

Original: 3. Multi-task: partition problem by protein
Revised: 3. M ULTI - TASK : PARTITION PROBLEM BY PROTEIN

Original: Two control experiments were run on the cyp dataset in a multitask setting, with the goal of predicting CP3A4 activity in the test set.
Revised: Two control experiments were run on the cyp dataset in a multi-task setting (Figure 2), with the goal of predicting CP3A4 activity in the test set.

Original: The first control consists in a single task model with only one global expert, the second in a multi-task model with also only one global expert.
Revised: The first control consists in a single task model with only one global expert, the second in a multi-task model with also only one global expert.

Original: These controls are compared to a multi-task model with both a global expert andprotein experts, one for each of the CYPs in the dataset (CP2D6, CP1A2, CP2C9, CP2CJ, CP3A4).
Revised: These controls are compared to a multi-task model with both a global expert and 5 protein experts, one for each of the proteins in the dataset (CP2D6, CP1A2, CP2C9, CP2CJ, CP3A4).

Original: Results are reported in Figure 4.
Revised: Results are reported in Figure 4.

Original: Predictions from the single task model are generally higher than from the multi-task model, and lower-activity molecules are not well predicted.
Revised: Predictions from the single task model are generally higher than from the multi-task model, and lower-activity molecules are not well predicted.

Original: The multi-task baseline pushes those predictions down but the range of predicted values stays narrow; adding protein experts improves the MSE in this particular instance.
Revised: The multi-task baseline pushes those predictions down but the range of predicted values stays narrow; adding protein experts improves the MSE in this particular instance.

Original: 3. Model introspection
Revised: 3. M ODEL INTROSPECTION

Original: The main interest of partitioning a QSAR problem isfor downstream model introspection.
Revised: An important advantage of partitioning a QSAR problem is downstream model introspection.

Original: In Figure5, the weights assigned to different local experts are shown for two molecules in the akt1 validation set.
Revised: In Figure 5, the weights assigned to different local experts are shown for two molecules in the akt validation set.

Original: If the model was trained correctly, most of the gate weights should go to experts specialised in the relevant chemical series and the global expert.
Revised: If the model was trained correctly, most of the gate weights should go to experts specialised in the relevant chemical series and the global expert.

Original: Molecules belonging to chemical series which are not covered by local experts are expected to be mostly assigned to the global expert.
Revised: Molecules belonging to chemical series which are not covered by local experts are expected to be mostly assigned to the global expert.

Original: The same type of visualization can be produced for the multi-task, multi-protein Glolloc architecture (Figure 2): in Figure 6, weights assigned to different protein experts are shown for two molecules in the cyp validation set.
Revised: The same type of visualization can be produced for the multi-task Glolloc architecture (Figure 2): in Figure 6, weights assigned to different protein experts are shown for two molecules in the cyp validation set.

Original: Only the gate weights for  CP3A4 activity prediction are shown in the figure, as different weights would be assigned if another task was queried (e.g. CP2D6, CP1A2, CP2C9, or CP2CJ).
Revised: The network predicts the activity against 5 proteins (CP3A4, CP2D6, CP1A2, CP2C9, CP2CJ), but only weights for the CP3A4 task are shown in the figure.

Original: For the CP3A4 task, most predictions are dominated by the CP3A4 expert and the global expert.
Revised: For that task, most predictions are dominated by the CP3A4 expert and the global expert.

Original:  Conclusion and next steps
Revised:  C ONCLUSION AND NEXT STEPS

Original: Glolloc adds local structure to QSAR models, which can be used for downstream model introspection by visualising how local or global structures contribute to the final prediction.
Revised: Glolloc adds local structure to QSAR models, which can be used for downstream model introspection by visualising how local or global structures contribute to the final prediction.

Original: In the preliminary investigations shown in this paper, models adding more structure, based on chemical series, clusters, or single proteins, can often perform better than controls (RF/single experts/random experts).
Revised: In the preliminary investigations shown in this paper, models adding more structure, based on chemical series, clusters, or single proteins, can often perform better than controls (RF/single experts/random experts).

Original: However, these models need more careful optimisation, and more datasets and splits should be investigated in follow-up experiments to build more confidence in performance improvement.
Revised: However, these models need more careful optimisation and more datasets and splits should be investigated in follow-up experiments to build more confidence in performance improvement.

Original: There is still progress to be made on the side of expert training, as currently the masked architecture is not ideal and needs for the user to monitor a series of losses.
Revised: There is still progress to be made on the side of expert training, as experts are defined using data masks and users have to monitor a series of losses.

Original: Arguably, the discrete, masked multi-expert setting could be replaced by a more continuous approach.
Revised: Arguably, the discrete, masked multi-expert setting could be replaced by a more continuous approach.

Original: Alternatively, an attention mechanism could be used instead of experts, to give more attention to predefined dataset structures.
Revised: Alternatively, an attention mechanism could be used instead of experts, to give more attention to predefined dataset structures.

Original: Different 2D and 3D depictions of molecules could also be used to define additional experts, as well as potential groupings of related tasks, such as protein families in the multi-task setting.
Revised: Different 2D and 3D depictions of molecules could also be used to define additional experts, as well as potential groupings of related tasks, such as protein families in the multi-task setting.

Original: Glolloc-like architectures could also be used to model different endpoint types or experimental settings.
Revised: Glolloc-like architectures could also be used to model different endpoint types or experimental settings

Revised:  A PPENDIX 5. D ESCRIPTORS

Revised: We used the rdkit (RDKit, online) implementation of Morgan fingerprints.

Original: Appendix 5. Other automated methods to define chemical series
Revised: 5. H YPERPARAMETER SEARCH

Revised: For all experiments, parameters were optimised on the valid set without cross-validation. For Random Forests, combinations of hyperparameters were systematically screened from maximum depth = { 5, 25, 50, 75, 100 } and number of estimators = { 100, 200 } . We used the scikit-learn (Pedregosa et al., 2011) implementation of Random Forests (Breiman, 2001) regression.

Revised: For Glolloc networks, hidden layer sizes were the only variable hyperparameter (two hidden layers with [64, 64] or [1024, 1024] units).

Revised: Only fully connected layers were used, with fixed parameters for dropout (0.25 for experts, 0.025 for the gate), early stopping patience (15), batch size (32), learning rate (0.0005), and temperature (1) for the gate softmax.

Revised: For the datasets in this paper, which are relatively small, the networks are easily trainable on a single CPU within a few minutes.

Revised: 5. G LOLLOC E XPERTS T RAINING

Revised: Expert losses are defined by the following equation:

Revised: [Equation]

Revised: Where expert i is a set of pairs of molecules and tasks { n, t } relevant to expert i , and the indicator function 1 ( { n , t }∈ expert i ) returns 1 if { n, t } is relevant to the i th expert, 0 if not.

Revised: In practice, experts are defined using masks, so that the losses for irrelevant data points are not taken into account during the training process: e.g. if an expert focuses on pyrazole-containing molecular compounds, all losses for molecules not containing a pyrazole substructure will be zeroed out.

Revised: Experts are independent, each having their own optimiser and loss.

Revised: This independence makes training the network challenging, as validation sets need to be carefully defined and loss curves monitored for each expert - local experts are in effect trained on subsets of the training data, and they are prone to overfitting.

Revised: This is only partially addressed by adding learning rate decay for each optimiser.

Revised: 5.3. T HE G ATE N ETWORK

Revised: The gate is a fully connected network with a softmax function, outputing a gate weight matrix of dimension [batch size, number of tasks, number of experts], the softmax function being applied on the last dimension, so that weights across experts sum to 1 for each task.

Original: Other methods of detecting chemical series in adata set can be usedbeyond tree-based, e.g. varying the number of molecules that can match an MCS in a data set, fragment-based methods, scaffold tree methods, or nearest neighbours algorithms.
Revised: Glolloc can use automated methods to define chemical series (”autosmarts”): tree-based, or neighbourhood based. They both rely on finding a set of maximum common substructures in the dataset, using hierarchical clustering or neighbourhood membership, respectively. We used the RDKit (RDKit, online) implementation of fuzzy MCS - with a minimum of 20 molecules per MCS.

Revised: The tree-based method (Algorithm 1) is similar to the one described recently by Kruger al (Kruger et al., 2020): building fuzzy maximum common substructures (MCS) from nodes in a tree obtained using an agglomerative clustering algorithm, going from the root to the leaves: when a series (MCS) is found in a parent node, children are discarded.

Revised: The difference between the Glolloc implementation and the one by Kruger et al. is the criterion to validate the MCS: they use a probability that a random molecule matches the scaffold, whereas Glolloc sets a minimum number of atoms for the scaffold, to guarantee a large enough scaffold (default value: 11 atoms).

Revised: Strict MCS ring matching rules and loose rules for atom and bond matching were arbitrarily used - the impact of different MCS algorithm settings is not explored in this paper.

Revised: min atoms ← user parameter ▷ minimum number of atoms in MCS min node size ← user parameter ▷ minimum number of molecules to construct an MCS from

Revised: T ← dendrogram ( D ) ▷ construct dendrogram from the molecule data matrix D

Revised: N ← node dict ( T )

Revised: ▷ dictionary mapping tree nodes to molecules

Revised: N ′ ← sort ( N ) ▷ sort nodes from root to leaves

Revised: MCS ← ∅ ▷ the MCS set stores the maximum common substructures for node, molecules in node in N ′ do if count ( molecules in node ) ≥ min node size then mcs ← fuzzy maximum substructure ( molecules in node )

Revised: ▷ if a large enough MCS is found in parent node if mcs exists and mcs.n atoms ≥ min atoms then

Revised: MCS ← MCS ∪ { mcs } set children of node in N ′ to ∅ ▷ discard children nodes end if end if end for

Revised: Other methods of detecting chemical series in a data set can be used beyond tree-based, e.g. varying the number of molecules that can match an MCS in a data set, fragment-based methods, scaffold tree methods, or nearest neighbours algorithms.

Original: The neighbourhood-based autosmarts method for series identification also implemented in Glolloc  is based on the algorithm described by Hattori et al. 
Revised: The neighbourhood-based autosmarts method for series identification also implemented in Glolloc (Algorithm 2) is based on the algorithm described by Hattori et al. (Hattori et al., 2008)

Original: (19) and adapted for MCS series definition.
Revised: and adapted for MCS series definition.

Original: Hattori et al.’s
Revised: Hattori et al.’s

Original: algorithm is based on iteratively finding representative molecules with the largest neighbourhood and removing them and their neighbours from the pool of molecules.
Revised: algorithm is based on iteratively finding representative molecules with the largest neighbourhood and removing them and their neighbours from the pool of molecules.

Original: This method is dependent on a similarity threshold parameter to define the neighbours.
Revised: This method is dependent on a similarity threshold parameter to define the neighbours.

Original: For Glolloc, neighbourhoods are also iteratively defined, but at each iteration a fuzzy MCS from the top neighbourhood is defined and all molecules matching that MCS are removed from the pool used at the next iteration.
Revised: For Glolloc, neighbourhoods are also iteratively defined, but at each iteration a fuzzy MCS from the top neighbourhood is found and all molecules matching that MCS are removed from the pool used at the next iteration.

Original: The neighbourhood-based approach is not used in this paper for benchmarks as it seemed to provide similar results to the tree-based method when varying the similarity threshold for our data sets.
Revised: The neighbourhood-based approach is not used in this paper for benchmarks as it seemed to provide similar results to the tree-based method when varying the similarity threshold for our data sets.

Original: Algorithm 2 Neighbourhood autosmarts algorithm: find fuzzy maximum common substructures (MCS) in different regions of the chemical space using nearest neighbours min_ similarity ← user parameter ▷ similarity cutoff min_ molecules ← user parameter ▷ minimum number of molecules per MCS bag_ of_ molecules ← set of all molecules indices
Revised: Algorithm 2 Neighbourhood autosmarts algorithm: find fuzzy maximum common substructures (MCS) in different regions of the chemical space using nearest neighbours min similarity ← user parameter ▷ similarity cutoff min molecules ← user parameter ▷ minimum number of molecules per MCS bag of molecules ← set of all molecules indices

Original: S ← similarity_ matrix ( D ) ▷ similarity matrix from molecule data matrix D
Revised: S ← similarity matrix ( D ) ▷ similarity matrix from molecule data matrix D

Original: MCS ← ∅ ▷ the MCS set stores the maximum common substructures while bag_ of_ molecule is not empty do ▷ find neighbourhood of molecule with max number of neighbours top_ neighbourhood ← f ( S, min_ similarity ) if count ( top_ neighbourhood ) ≥ min_ molecules then mcs ← fuzzy_ maximum_ substructure ( top_ neighbourhood) if mcs exists then
Revised: MCS ← ∅ ▷ the MCS set stores the maximum common substructures while bag of molecule is not empty do ▷ find neighbourhood of molecule with max number of neighbours top neighbourhood ← f ( S, min similarity ) if count ( top neighbourhood ) ≥ min molecules then mcs ← fuzzy maximum substructure ( top neighbourhood

Original:  ▷ if MCS is found in neighbourhood 
Revised: ) if mcs exists then ▷ if MCS is found in neighbourhood MCS ← MCS

Original: MCS ← MCS ∪ { mcs } mcs_ matches ← matches ( bag_ of_ molecule, mcs )
Revised: ∪ { mcs } mcs matches ← matches ( bag of molecule, mcs )

Original: ▷ discard molecules matching MCS for next iteration bag_ of_ molecule ← bag_ of_ molecule \ mcs_ matches 
Revised: ▷ discard molecules matching MCS for next iteration bag of molecule ← bag of molecule \ mcs matches S

Original: S [ mcs_ matches ] ←0 .
Revised: mcs matches ] ←

Original: ▷ set similarities of discarded molecules toend if end if end while
Revised: ▷ set similarities of discarded molecules toend if end if end while