Original: G LOLLOC : M IXTURE OF G LOBAL AND L OCAL E XPERTS FOR M OLECULAR A CTIVITY P REDICTION
Revised: G LOLLOC : M IXTURE OF G LOBAL AND L OCAL E XPERTS FOR M OLECULAR A CTIVITY P REDICTION

Original: A BSTRACT
Revised: A BSTRACT

Original: Quantitative structure-activity relationships (QSAR) models have been used for decades to predict the activity of small molecules, using encodings of the molecular structure, for which simple 2D descriptors of the molecular graph are still most commonly used.
Revised: Quantitative structure-activity relationships (QSAR) models have been used for decades to predict the activity of small molecules, using encodings of the molecular structure, for which simple 2D descriptors of the molecular graph are still most commonly used.

Original: One of the recurrent problems of QSAR is that relationships observed for a specific scaffold (pruned molecular skeleton) are often not transferable to another; this is often addressed by building several local models from subsets of the chemical space.
Revised: One of the recurrent problems of QSAR is that relationships observed for a specific scaffold (pruned molecular skeleton) are often not transferable to another; this is often addressed by building several local models from subsets of the chemical space.

Original: Similarly, single task models sometimes outperform large multi-task models in predicting the activity of small molecules against specific proteins.
Revised: Similarly, single task models sometimes outperform large multi-task models in predicting the activity of small molecules against specific proteins.

Original: In this paper, we introduce Glolloc, a global-local MoE-QSAR architecture, based on a Mixture of Experts (MoE) framework.
Revised: In this paper, we introduce Glolloc, a global-local MoE-QSAR architecture, based on a Mixture of Experts (MoE) framework.

Original: Glolloc combines predictions from global and local experts, provides a built-in model introspection tool, can enhance model performance, and removes the need to maintain several local models.
Revised: Glolloc combines predictions from global and local experts, provides a built-in model introspection tool, can enhance model performance, and removes the need to maintain several local models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Mixtures of experts (MoE) are a subset of ensemble learning algorithms, consisting in individual learners or experts, and a gate that assigns weights to different learners.
Revised: Mixtures of experts (MoE) are a subset of ensemble learning algorithms, consisting in individual learners or experts, and a gate that assigns weights to different learners.

Original: They were relatively popular in the 90s (Jordan & Jacobs, 1993; Jacobs et al., 1991), and have recently regained interest in the deep learning community, notably for language and vision (Ruiz et al., 2021;
Revised: They were relatively popular in the 90s (Jordan & Jacobs, 1993; Jacobs et al., 1991), and have recently regained interest in the deep learning community, notably for language and vision (Ruiz et al., 2021;

Original: Shazeer et al., 2017).
Revised: Shazeer et al., 2017).

Original: MoE can be seen as a type of meta-learning approach as it ”learns to learn” from an ensemble of experts, which can be explicitly pre-defined e.g. by using clustering or subsets of input features (Gutta et al., 2000; Tang et al., 2002).
Revised: MoE can be seen as a type of meta-learning approach as it ”learns to learn” from an ensemble of experts, which can be explicitly pre-defined e.g. by using clustering or subsets of input features (Gutta et al., 2000; Tang et al., 2002).

Original: In quantitative structure-activity relationships (QSAR) models, where the aim is to predict activity from molecular structures (e.g. activity against a protein of interest), local models trained on subsets of molecules often provide better performances than global models trained on the whole dataset (Yuan et al., 2006).
Revised: In quantitative structure-activity relationships (QSAR) models, where the aim is to predict activity from molecular structures (e.g. activity against a protein of interest), local models trained on subsets of molecules often provide better performances than global models trained on the whole dataset (Yuan et al., 2006).

Original: In drug discovery projects, a local model is often constructed for each molecular series of interest (set of molecules with a similar skeleton or ”scaffold”).
Revised: In drug discovery projects, a local model is often constructed for each molecular series of interest (set of molecules with a similar skeleton or ”scaffold”).

Original: At a given stage, several chemical series might be investigated, and several models are separately optimised and maintained.
Revised: At a given stage, several chemical series might be investigated, and several models are separately optimised and maintained.

Original: However, these local models discard potentially valuable information, and do not always perform better than global models.
Revised: However, these local models discard potentially valuable information, and do not always perform better than global models.

Original: Furthermore, some molecules share characteristics from different series and would benefit from combining several local model predictions.
Revised: Furthermore, some molecules share characteristics from different series and would benefit from combining several local model predictions.

Original: Methods that are able to combine advantages from both local and global models should therefore be of particular interest in the field.
Revised: Methods that are able to combine advantages from both local and global models should therefore be of particular interest in the field.

Original: We thought a MoE architecture could be a way to combine predictions from global and local experts, each local expert being specialised in a chemical series of interest.
Revised: We thought a MoE architecture could be a way to combine predictions from global and local experts, each local expert being specialised in a chemical series of interest.

Original: Local experts for QSAR can also be defined in other ways.
Revised: Local experts for QSAR can also be defined in other ways.

Original: When the goal is to predict the activity of a small molecule against several proteins at the same time, one local expert can be assigned to each protein task.
Revised: When the goal is to predict the activity of a small molecule against several proteins at the same time, one local expert can be assigned to each protein task.

Original: Multi-protein models can perform better or worse than single task learning, depending on the protein (Gaspar et al., 2021) - subdividing the problem and training one expert per protein or protein family, in addition to a global expert trained on everything, could help to better adapt predictions to specific proteins.
Revised: Multi-protein models can perform better or worse than single task learning, depending on the protein (Gaspar et al., 2021) - subdividing the problem and training one expert per protein or protein family, in addition to a global expert trained on everything, could help to better adapt predictions to specific proteins.

Original: The MoE literature for QSAR is very scarce - MoEs were recently investigated (D¨org˝o et al., 2020) as a QSAR ensemble method across different feature spaces.
Revised: The MoE literature for QSAR is very scarce - MoEs were recently investigated (D¨org˝o et al., 2020) as a QSAR ensemble method across different feature spaces.

Original: In methods introduced in the following sections, the problem is naively partitioned by input instances rather than input features, although combining both approaches would be an interesting follow-up.
Revised: In methods introduced in the following sections, the problem is naively partitioned by input instances rather than input features, although combining both approaches would be an interesting follow-up.

Original: Partitioning the input space based on chemistry expert knowledge, and then using a MoE architecture to choose between local and global experts has not yet been attempted.
Revised: Partitioning the input space based on chemistry expert knowledge, and then using a MoE architecture to choose between local and global experts has not yet been attempted.

Original: The main aim of the following experiments is to explore different ways to partition the problem in a chemically relevant way, and come up with a model able to focus on local aspects whilst not discarding valuable information.
Revised: The main aim of the following experiments is to explore different ways to partition the problem in a chemically relevant way, and come up with a model able to focus on local aspects whilst not discarding valuable information.

Original: The main contribution of this paper is  global-local (”Glolloc”) MoE-QSAR architecture for single or multi-task learning that: (a) combines advantages from both local and global models, which are routinely used separately in QSAR pipelines, (b) is structured using expert knowledge (e.g. medicinal chemist partitioning the space by chemical series), (c) removes the need to maintain many local models, (d) determines which experts are more appropriate for a given molecule, (e) provides a built-in model introspection tool via weights given to experts, and (f) can improve performance over a single global model or an ensemble of random experts.
Revised: The main contribution of this paper is a global-local (”Glolloc”) MoE-QSAR architecture for single or multi-task learning that: (a) combines advantages from both local and global models, which are routinely used separately in QSAR pipelines, (b) is structured using expert knowledge (e.g. medicinal chemist partitioning the space by chemical series), (c) removes the need to maintain many local models, (d) determines which experts are more appropriate for a given molecule, (e) provides a built-in model introspection tool via weights given to experts, and (f) can improve performance over a single global model or an ensemble of random experts.

Original:  M ETHODS 2. D ATA
Revised:  M ETHODS 2. D ATA

Original: [Table]
Revised: [Table]

Original: In the single task experiments, the 7 datasets (Table 1) correspond to 7 diverse proteins from the diverse subset of the DUD-E data collection (Mysinger et al., 2012).
Revised: In the single task experiments, the 7 datasets (Table 1) correspond to 7 diverse proteins from the diverse subset of the DUD-E data collection (Mysinger et al., 2012).

Original: Note that whilst the diverse subset from DUD-E was used to choose the proteins, the ligand data was retrieved from a publicly available benchmark dataset (Lenselink et al., 2017b) curated by Lenselink et al.
Revised: Note that whilst the diverse subset from DUD-E was used to choose the proteins, the ligand data was retrieved from a publicly available benchmark dataset (Lenselink et al., 2017b) curated by Lenselink et al.

Original: (Lenselink et al., 2017a), which compiles activities of small molecules extracted from the ChEMBL database (Gaulton et al., 2016).
Revised: (Lenselink et al., 2017a), which compiles activities of small molecules extracted from the ChEMBL database (Gaulton et al., 2016).

Original: For our purposes, instances are small molecules, and the endpoint is pChEMBL activity (-
Revised: For our purposes, instances are small molecules, and the endpoint is pChEMBL activity (-

Original: Log(molar IC50, XC50, EC50, AC50, Ki, Kd or Potency)).
Revised: Log(molar IC50, XC50, EC50, AC50, Ki, Kd or Potency)).

Original: For each of the 7 datasets, a tree-based ”autosmarts” algorithm
Revised: For each of the 7 datasets, a tree-based ”autosmarts” algorithm

Original: (Algorithm 1) was used on molecules with pChEMBL value > 5.5 to find a top chemical series of interest - its corresponding SMARTS pattern visualisation is reported in
Revised: (Algorithm 1) was used on molecules with pChEMBL value > 5.5 to find a top chemical series of interest - its corresponding SMARTS pattern visualisation is reported in

Original: Table 1.
Revised: Table 1.

Original: Compounds matching that pattern were split into train, valid and test using time splits.
Revised: Compounds matching that pattern were split into train, valid and test using time splits.

Original: This train set was augmented with data not matching the pattern, resulting in a train set that contains both the local data and other diverse molecules.
Revised: This train set was augmented with data not matching the pattern, resulting in a train set that contains both the local data and other diverse molecules.

Original: The time splits were defined using publication year, to mimic what happens in a drug discovery project, where the aim is to do better on more recent data.
Revised: The time splits were defined using publication year, to mimic what happens in a drug discovery project, where the aim is to do better on more recent data.

Original: It should be noted that publication year is not an ideal proxy and does not reflect the actual experiment date.
Revised: It should be noted that publication year is not an ideal proxy and does not reflect the actual experiment date.

Original: The cp3a4 dataset was supplemented by other cytochrome data for the multi-task challenge (CP2D6,
Revised: The cp3a4 dataset was supplemented by other cytochrome data for the multi-task challenge (CP2D6,

Original: CP1A2, CP2C9, and CP2CJ) to construct the cyp dataset - the test dataset being the same as for cp3a4.
Revised: CP1A2, CP2C9, and CP2CJ) to construct the cyp dataset - the test dataset being the same as for cp3a4.

Original: ECFP4 2048 descriptors were used to featurise molecules: Morgan fingerprints with a radius ofand size 2048.
Revised: ECFP4 2048 descriptors were used to featurise molecules: Morgan fingerprints with a radius ofand size 2048.

Original: 2. P ROPOSED SINGLE TASK AND MULTI - TASK NETWORKS
Revised: 2. P ROPOSED SINGLE TASK AND MULTI - TASK NETWORKS

Original: Two architectures were investigated: a single-task network with local experts specialising in different regions of the chemical space (Figure 1), and a multi-task network to predict the activity of multiple proteins, where local experts specialise in predicting specific proteins (Figure 2).
Revised: Two architectures were investigated: a single-task network with local experts specialising in different regions of the chemical space (Figure 1), and a multi-task network to predict the activity of multiple proteins, where local experts specialise in predicting specific proteins (Figure 2).

Original: In each case, in addition to the local networks (chemical series or protein experts), a global expert is trained on the whole data set, and is expected to perform better for molecules outside of the applicability domain of the local models.
Revised: In each case, in addition to the local networks (chemical series or protein experts), a global expert is trained on the whole data set, and is expected to perform better for molecules outside of the applicability domain of the local models.

Original: The gate network, trained at the same time as the experts, assigns weights to each of them.
Revised: The gate network, trained at the same time as the experts, assigns weights to each of them.

Original: The final predicted activity for a molecule n and task t is the weighted sum of outputs from different experts ( O ) and gate weights ( G ) (Equation 1).
Revised: The final predicted activity for a molecule n and task t is the weighted sum of outputs from different experts ( O ) and gate weights ( G ) (Equation 1).

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2. A UTOMATED METHODS TO DEFINE CHEMICAL SERIES
Revised: 2. A UTOMATED METHODS TO DEFINE CHEMICAL SERIES

Original: In a drug discovery project, chemical series are typically identified by medicinal chemists.
Revised: In a drug discovery project, chemical series are typically identified by medicinal chemists.

Original: Chemoinformaticians encode them as SMARTS - a type of regex language, which allows to query molecules containing specific substructures.
Revised: Chemoinformaticians encode them as SMARTS - a type of regex language, which allows to query molecules containing specific substructures.

Original: However, for benchmarks on public data sets, an automated way to partition the chemical space is needed.
Revised: However, for benchmarks on public data sets, an automated way to partition the chemical space is needed.

Original: Note that in a real-world scenario, project chemists would help in the definition of the series to add their domain knowledge to the model structure.
Revised: Note that in a real-world scenario, project chemists would help in the definition of the series to add their domain knowledge to the model structure.

Original: Glolloc implements 3 methods to generate chemical series experts from a chemical dataset: cluster-based, tree-based, and neighbourhoodbased.
Revised: Glolloc implements 3 methods to generate chemical series experts from a chemical dataset: cluster-based, tree-based, and neighbourhoodbased.

Original: The cluster-based method is self-explanatory: a k-means clusterer using scikit-learn’s implementation (Pedregosa et al., 2011) is trained on the training set, and cluster labels are used to construct cluster-specific experts.
Revised: The cluster-based method is self-explanatory: a k-means clusterer using scikit-learn’s implementation (Pedregosa et al., 2011) is trained on the training set, and cluster labels are used to construct cluster-specific experts.

Original: The tree-based and neighbourhood-based methods both rely on finding fuzzy maximum substructures (MCS) in groups of molecules.
Revised: The tree-based and neighbourhood-based methods both rely on finding fuzzy maximum substructures (MCS) in groups of molecules.

Original: They are described in further detail in the Appendix.
Revised: They are described in further detail in the Appendix.

Original: We used the tree-based method in this paper to define our chemical series, as we obtained similar results with the neighbourhood-based method.
Revised: We used the tree-based method in this paper to define our chemical series, as we obtained similar results with the neighbourhood-based method.

Original: The advantage of MCS-based methods is the automatically generated SMARTS patterns that can be visualised and interpreted easily - this type of approach is therefore coined ”autosmarts” in this paper.
Revised: The advantage of MCS-based methods is the automatically generated SMARTS patterns that can be visualised and interpreted easily - this type of approach is therefore coined ”autosmarts” in this paper.

Original: [Table]
Revised: [Table]

Original:  R ESULTS 3. S INGLE TASK : PARTITION PROBLEM BY CHEMICAL SERIES
Revised:  R ESULTS 3. S INGLE TASK : PARTITION PROBLEM BY CHEMICAL SERIES

Original: This section compares Random Forests and Glolloc architectures using different partitions of the chemical space to define experts (Figure 1).
Revised: This section compares Random Forests and Glolloc architectures using different partitions of the chemical space to define experts (Figure 1).

Original: Two control experiments were designed for the Glolloc framework: a model using a single global expert, and a model using one global expert and 6 random experts.
Revised: Two control experiments were designed for the Glolloc framework: a model using a single global expert, and a model using one global expert and 6 random experts.

Original: The random experts divide the chemical space into 6 random sections.
Revised: The random experts divide the chemical space into 6 random sections.

Original: The control experiments are run against two structured
Revised: The control experiments are run against two structured

Original: Glolloc models: one with the global expert and 6 cluster experts, the other with the global expert and autosmarts (Algorithm 1) experts.
Revised: Glolloc models: one with the global expert and 6 cluster experts, the other with the global expert and autosmarts (Algorithm 1) experts.

Original: The more structured experts (based on clustering or autosmarts algorithms) yield lower mean squared errors (MSE) for 5 out of 7 datasets, and higher Spearman rank correlation in 6 out of 7 (Table 2).
Revised: The more structured experts (based on clustering or autosmarts algorithms) yield lower mean squared errors (MSE) for 5 out of 7 datasets, and higher Spearman rank correlation in 6 out of 7 (Table 2).

Original: When comparing the best glolloc expert combination (whether random or structured) against a single expert or Random Forests (Figure 3), Random Forests wins only for the gcr dataset and has the worst performance in all other cases.
Revised: When comparing the best glolloc expert combination (whether random or structured) against a single expert or Random Forests (Figure 3), Random Forests wins only for the gcr dataset and has the worst performance in all other cases.

Original: 3. M ULTI - TASK : PARTITION PROBLEM BY PROTEIN
Revised: 3. M ULTI - TASK : PARTITION PROBLEM BY PROTEIN

Original: Two control experiments were run on the cyp dataset in a multi-task setting (Figure 2), with the goal of predicting CP3A4 activity in the test set.
Revised: Two control experiments were run on the cyp dataset in a multi-task setting (Figure 2), with the goal of predicting CP3A4 activity in the test set.

Original: The first control consists in a single task model with only one global expert, the second in a multi-task model with also only one global expert.
Revised: The first control consists in a single task model with only one global expert, the second in a multi-task model with also only one global expert.

Original: These controls are compared to a multi-task model with both a global expert and 5 protein experts, one for each of the proteins in the dataset (CP2D6, CP1A2, CP2C9, CP2CJ, CP3A4).
Revised: These controls are compared to a multi-task model with both a global expert and 5 protein experts, one for each of the proteins in the dataset (CP2D6, CP1A2, CP2C9, CP2CJ, CP3A4).

Original: Results are reported in Figure 4.
Revised: Results are reported in Figure 4.

Original: Predictions from the single task model are generally higher than from the multi-task model, and lower-activity molecules are not well predicted.
Revised: Predictions from the single task model are generally higher than from the multi-task model, and lower-activity molecules are not well predicted.

Original: The multi-task baseline pushes those predictions down but the range of predicted values stays narrow; adding protein experts improves the MSE in this particular instance.
Revised: The multi-task baseline pushes those predictions down but the range of predicted values stays narrow; adding protein experts improves the MSE in this particular instance.

Original: 3. M ODEL INTROSPECTION
Revised: 3. M ODEL INTROSPECTION

Original: An important advantage of partitioning a QSAR problem is downstream model introspection.
Revised: An important advantage of partitioning a QSAR problem is downstream model introspection.

Original: In
Revised: In

Original: Figure 5, the weights assigned to different local experts are shown for two molecules in the akt validation set.
Revised: Figure 5, the weights assigned to different local experts are shown for two molecules in the akt validation set.

Original: If the model was trained correctly, most of the gate weights should go to experts specialised in the relevant chemical series and the global expert.
Revised: If the model was trained correctly, most of the gate weights should go to experts specialised in the relevant chemical series and the global expert.

Original: Molecules belonging to chemical series which are not covered by local experts are expected to be mostly assigned to the global expert.
Revised: Molecules belonging to chemical series which are not covered by local experts are expected to be mostly assigned to the global expert.

Original: The same type of visualization can be produced for the multi-task Glolloc architecture (Figure 2): in Figure 6, weights assigned to different protein experts are shown for two molecules in the cyp validation set.
Revised: The same type of visualization can be produced for the multi-task Glolloc architecture (Figure 2): in Figure 6, weights assigned to different protein experts are shown for two molecules in the cyp validation set.

Original: The network predicts the activity against 5 proteins (CP3A4, CP2D6, CP1A2, CP2C9,
Revised: The network predicts the activity against 5 proteins (CP3A4, CP2D6, CP1A2, CP2C9,

Original: CP2CJ), but only weights for the CP3A4 task are shown in the figure.
Revised: CP2CJ), but only weights for the CP3A4 task are shown in the figure.

Original: For that task, most predictions are dominated by the CP3A4 expert and the global expert.
Revised: For that task, most predictions are dominated by the CP3A4 expert and the global expert.

Original:  C ONCLUSION AND NEXT STEPS
Revised:  C ONCLUSION AND NEXT STEPS

Original: Glolloc adds local structure to QSAR models, which can be used for downstream model introspection by visualising how local or global structures contribute to the final prediction.
Revised: Glolloc adds local structure to QSAR models, which can be used for downstream model introspection by visualising how local or global structures contribute to the final prediction.

Original: In the preliminary investigations shown in this paper, models adding more structure, based on chemical series, clusters, or single proteins, can often perform better than controls (RF/single experts/random experts).
Revised: In the preliminary investigations shown in this paper, models adding more structure, based on chemical series, clusters, or single proteins, can often perform better than controls (RF/single experts/random experts).

Original: However, these models need more careful optimisation and more datasets and splits should be investigated in follow-up experiments to build more confidence in performance improvement.
Revised: However, these models need more careful optimisation and more datasets and splits should be investigated in follow-up experiments to build more confidence in performance improvement.

Original: There is still progress to be made on the side of expert training, as experts are defined using data masks and users have to monitor a series of losses.
Revised: There is still progress to be made on the side of expert training, as experts are defined using data masks and users have to monitor a series of losses.

Original: Arguably, the discrete, masked multi-expert setting could be replaced by a more continuous approach.
Revised: Arguably, the discrete, masked multi-expert setting could be replaced by a more continuous approach.

Original: Alternatively, an attention mechanism could be used instead of experts, to give more attention to predefined dataset structures.
Revised: Alternatively, an attention mechanism could be used instead of experts, to give more attention to predefined dataset structures.

Original: Different 2D and 3D depictions of molecules could also be used to define additional experts, as well as potential groupings of related tasks, such as protein families in the multi-task setting.
Revised: Different 2D and 3D depictions of molecules could also be used to define additional experts, as well as potential groupings of related tasks, such as protein families in the multi-task setting.

Original: Glolloc-like architectures could also be used to model different endpoint types or experimental settings
Revised: Glolloc-like architectures could also be used to model different endpoint types or experimental settings

Original:  A PPENDIX 5. D ESCRIPTORS
Revised:  A PPENDIX 5. D ESCRIPTORS

Original: We used the rdkit (RDKit, online) implementation of Morgan fingerprints.
Revised: We used the rdkit (RDKit, online) implementation of Morgan fingerprints.

Original: 5. H YPERPARAMETER SEARCH
Revised: 5. H YPERPARAMETER SEARCH

Original: For all experiments, parameters were optimised on the valid set without cross-validation.
Revised: For all experiments, parameters were optimised on the valid set without cross-validation.

Original: For Random Forests, combinations of hyperparameters were systematically screened from maximum depth = { 5, 25, 50, 75, 100 } and number of estimators = { 100, 200 } .
Revised: For Random Forests, combinations of hyperparameters were systematically screened from maximum depth = { 5, 25, 50, 75, 100 } and number of estimators = { 100, 200 } .

Original: We used the scikit-learn (Pedregosa et al., 2011) implementation of Random Forests (Breiman, 2001) regression.
Revised: We used the scikit-learn (Pedregosa et al., 2011) implementation of Random Forests (Breiman, 2001) regression.

Original: For Glolloc networks, hidden layer sizes were the only variable hyperparameter (two hidden layers with [64, 64] or [1024, 1024] units).
Revised: For Glolloc networks, hidden layer sizes were the only variable hyperparameter (two hidden layers with [64, 64] or [1024, 1024] units).

Original: Only fully connected layers were used, with fixed parameters for dropout (0.25 for experts, 0.025 for the gate), early stopping patience (15), batch size (32), learning rate (0.0005), and temperature (1) for the gate softmax.
Revised: Only fully connected layers were used, with fixed parameters for dropout (0.25 for experts, 0.025 for the gate), early stopping patience (15), batch size (32), learning rate (0.0005), and temperature (1) for the gate softmax.

Original: For the datasets in this paper, which are relatively small, the networks are easily trainable on a single CPU within a few minutes.
Revised: For the datasets in this paper, which are relatively small, the networks are easily trainable on a single CPU within a few minutes.

Original: 5. G LOLLOC E XPERTS T RAINING
Revised: 5. G LOLLOC E XPERTS T RAINING

Original: Expert losses are defined by the following equation:
Revised: Expert losses are defined by the following equation:

Original: [Equation]
Revised: [Equation]

Original: Where expert i is a set of pairs of molecules and tasks { n, t } relevant to expert i , and the indicator function 1 ( { n , t }∈ expert i ) returns 1 if { n, t } is relevant to the i th expert, 0 if not.
Revised: Where expert i is a set of pairs of molecules and tasks { n, t } relevant to expert i , and the indicator function 1 ( { n , t }∈ expert i ) returns 1 if { n, t } is relevant to the i th expert, 0 if not.

Original: In practice, experts are defined using masks, so that the losses for irrelevant data points are not taken into account during the training process: e.g. if an expert focuses on pyrazole-containing molecular compounds, all losses for molecules not containing a pyrazole substructure will be zeroed out.
Revised: In practice, experts are defined using masks, so that the losses for irrelevant data points are not taken into account during the training process: e.g. if an expert focuses on pyrazole-containing molecular compounds, all losses for molecules not containing a pyrazole substructure will be zeroed out.

Original: Experts are independent, each having their own optimiser and loss.
Revised: Experts are independent, each having their own optimiser and loss.

Original: This independence makes training the network challenging, as validation sets need to be carefully defined and loss curves monitored for each expert - local experts are in effect trained on subsets of the training data, and they are prone to overfitting.
Revised: This independence makes training the network challenging, as validation sets need to be carefully defined and loss curves monitored for each expert - local experts are in effect trained on subsets of the training data, and they are prone to overfitting.

Original: This is only partially addressed by adding learning rate decay for each optimiser.
Revised: This is only partially addressed by adding learning rate decay for each optimiser.

Original: 5.3. T HE G ATE N ETWORK
Revised: 5.3. T HE G ATE N ETWORK

Original: The gate is a fully connected network with a softmax function, outputing a gate weight matrix of dimension [batch size, number of tasks, number of experts], the softmax function being applied on the last dimension, so that weights across experts sum to 1 for each task.
Revised: The gate is a fully connected network with a softmax function, outputing a gate weight matrix of dimension [batch size, number of tasks, number of experts], the softmax function being applied on the last dimension, so that weights across experts sum to 1 for each task.

Original: 5. A UTOMATED M ETHODS TO D EFINE C HEMICAL S ERIES
Revised: 5. A UTOMATED M ETHODS TO D EFINE C HEMICAL S ERIES

Original: Glolloc can use automated methods to define chemical series (”autosmarts”): tree-based, or neighbourhood based.
Revised: Glolloc can use automated methods to define chemical series (”autosmarts”): tree-based, or neighbourhood based.

Original: They both rely on finding a set of maximum common substructures in the dataset, using hierarchical clustering or neighbourhood membership, respectively.
Revised: They both rely on finding a set of maximum common substructures in the dataset, using hierarchical clustering or neighbourhood membership, respectively.

Original: We used the RDKit (RDKit, online) implementation of fuzzy MCS - with a minimum of 20 molecules per MCS.
Revised: We used the RDKit (RDKit, online) implementation of fuzzy MCS - with a minimum of 20 molecules per MCS.

Original: The tree-based method (Algorithm 1) is similar to the one described recently by Kruger al (Kruger et al., 2020): building fuzzy maximum common substructures (MCS) from nodes in a tree obtained using an agglomerative clustering algorithm, going from the root to the leaves: when a series (MCS) is found in a parent node, children are discarded.
Revised: The tree-based method (Algorithm 1) is similar to the one described recently by Kruger al (Kruger et al., 2020): building fuzzy maximum common substructures (MCS) from nodes in a tree obtained using an agglomerative clustering algorithm, going from the root to the leaves: when a series (MCS) is found in a parent node, children are discarded.

Original: The difference between the Glolloc implementation and the one by Kruger et al. is the criterion to validate the MCS: they use a probability that a random molecule matches the scaffold, whereas Glolloc sets a minimum number of atoms for the scaffold, to guarantee a large enough scaffold (default value: 11 atoms).
Revised: The difference between the Glolloc implementation and the one by Kruger et al. is the criterion to validate the MCS: they use a probability that a random molecule matches the scaffold, whereas Glolloc sets a minimum number of atoms for the scaffold, to guarantee a large enough scaffold (default value: 11 atoms).

Original: Strict MCS ring matching rules and loose rules for atom and bond matching were arbitrarily used - the impact of different MCS algorithm settings is not explored in this paper.
Revised: Strict MCS ring matching rules and loose rules for atom and bond matching were arbitrarily used - the impact of different MCS algorithm settings is not explored in this paper.

Original: min atoms ← user parameter ▷ minimum number of atoms in MCS min node size ← user parameter ▷ minimum number of molecules to construct an MCS from
Revised: min atoms ← user parameter ▷ minimum number of atoms in MCS min node size ← user parameter ▷ minimum number of molecules to construct an MCS from

Original: T ← dendrogram ( D ) ▷ construct dendrogram from the molecule data matrix D
Revised: T ← dendrogram ( D ) ▷ construct dendrogram from the molecule data matrix D

Original: N ← node dict ( T )
Revised: N ← node dict ( T )

Original: ▷ dictionary mapping tree nodes to molecules
Revised: ▷ dictionary mapping tree nodes to molecules

Original: N ′ ← sort ( N ) ▷ sort nodes from root to leaves
Revised: N ′ ← sort ( N ) ▷ sort nodes from root to leaves

Original: MCS ← ∅ ▷ the MCS set stores the maximum common substructures for node, molecules in node in N ′ do if count ( molecules in node ) ≥ min node size then mcs ← fuzzy maximum substructure ( molecules in node )
Revised: MCS ← ∅ ▷ the MCS set stores the maximum common substructures for node, molecules in node in N ′ do if count ( molecules in node ) ≥ min node size then mcs ← fuzzy maximum substructure ( molecules in node )

Original: ▷ if a large enough MCS is found in parent node if mcs exists and mcs.n atoms ≥ min atoms then
Revised: ▷ if a large enough MCS is found in parent node if mcs exists and mcs.n atoms ≥ min atoms then

Original: MCS ← MCS ∪ { mcs } set children of node in N ′ to ∅ ▷ discard children nodes end if end if end for
Revised: MCS ← MCS ∪ { mcs } set children of node in N ′ to ∅ ▷ discard children nodes end if end if end for

Original: Other methods of detecting chemical series in a data set can be used beyond tree-based, e.g. varying the number of molecules that can match an MCS in a data set, fragment-based methods, scaffold tree methods, or nearest neighbours algorithms.
Revised: Other methods of detecting chemical series in a data set can be used beyond tree-based, e.g. varying the number of molecules that can match an MCS in a data set, fragment-based methods, scaffold tree methods, or nearest neighbours algorithms.

Original: The neighbourhood-based autosmarts method for series identification also implemented in Glolloc (Algorithm 2) is based on the algorithm described by
Revised: The neighbourhood-based autosmarts method for series identification also implemented in Glolloc (Algorithm 2) is based on the algorithm described by

Original: Hattori et al.
Revised: Hattori et al.

Original: (Hattori et al., 2008)
Revised: (Hattori et al., 2008)

Original: and adapted for MCS series definition.
Revised: and adapted for MCS series definition.

Original: Hattori et al.’s
Revised: Hattori et al.’s

Original: algorithm is based on iteratively finding representative molecules with the largest neighbourhood and removing them and their neighbours from the pool of molecules.
Revised: algorithm is based on iteratively finding representative molecules with the largest neighbourhood and removing them and their neighbours from the pool of molecules.

Original: This method is dependent on a similarity threshold parameter to define the neighbours.
Revised: This method is dependent on a similarity threshold parameter to define the neighbours.

Original: For Glolloc, neighbourhoods are also iteratively defined, but at each iteration a fuzzy MCS from the top neighbourhood is found and all molecules matching that MCS are removed from the pool used at the next iteration.
Revised: For Glolloc, neighbourhoods are also iteratively defined, but at each iteration a fuzzy MCS from the top neighbourhood is found and all molecules matching that MCS are removed from the pool used at the next iteration.

Original: The neighbourhood-based approach is not used in this paper for benchmarks as it seemed to provide similar results to the tree-based method when varying the similarity threshold for our data sets.
Revised: The neighbourhood-based approach is not used in this paper for benchmarks as it seemed to provide similar results to the tree-based method when varying the similarity threshold for our data sets.

Original: Algorithm 2 Neighbourhood autosmarts algorithm: find fuzzy maximum common substructures (MCS) in different regions of the chemical space using nearest neighbours min similarity ← user parameter ▷ similarity cutoff min molecules ← user parameter ▷ minimum number of molecules per MCS bag of molecules
Revised: Algorithm 2 Neighbourhood autosmarts algorithm: find fuzzy maximum common substructures (MCS) in different regions of the chemical space using nearest neighbours min similarity ← user parameter ▷ similarity cutoff min molecules ← user parameter ▷ minimum number of molecules per MCS bag of molecules

Original: ← set of all molecules indices
Revised: ← set of all molecules indices

Original: S ← similarity matrix ( D ) ▷ similarity matrix from molecule data matrix D
Revised: S ← similarity matrix ( D ) ▷ similarity matrix from molecule data matrix D

Original: MCS ← ∅ ▷ the MCS set stores the maximum common substructures while bag of molecule is not empty do ▷ find neighbourhood of molecule with max number of neighbours top neighbourhood ← f ( S, min similarity ) if count ( top neighbourhood ) ≥ min molecules then mcs
Revised: MCS ← ∅ ▷ the MCS set stores the maximum common substructures while bag of molecule is not empty do ▷ find neighbourhood of molecule with max number of neighbours top neighbourhood ← f ( S, min similarity ) if count ( top neighbourhood ) ≥ min molecules then mcs

Original: ← fuzzy maximum substructure ( top neighbourhood
Revised: ← fuzzy maximum substructure ( top neighbourhood

Original: ) if mcs exists then ▷ if MCS is found in neighbourhood
Revised: ) if mcs exists then ▷ if MCS is found in neighbourhood

Original: MCS ←
Revised: MCS ←

Original: MCS
Revised: MCS

Original: ∪ {
Revised: ∪ {

Original: mcs
Revised: mcs

Original: } mcs
Revised: } mcs

Original: matches ← matches ( bag of molecule, mcs )
Revised: matches ← matches ( bag of molecule, mcs )

Original: ▷ discard molecules matching MCS for next iteration bag of molecule ← bag of molecule \ mcs matches S
Revised: ▷ discard molecules matching MCS for next iteration bag of molecule ← bag of molecule \ mcs matches S

Original: mcs matches ] ←
Revised: mcs matches ] ←

Original: ▷ set similarities of discarded molecules toend if end if end while
Revised: ▷ set similarities of discarded molecules toend if end if end while