Standardization of chemical compounds using language modeling
Abstract
With the growing amount of chemical data stored digitally, it has become crucial to represent chemical compounds consistently.
Harmonized representations facilitate the extraction of insightful information from datasets, and are advantageous for machine learning applications.
Compound standardization is typically accomplished using rule-based algorithms that modify undesirable descriptions of functional groups, resulting in a consistent representation throughout the dataset.
Here, we present the ﬁrst deep-learning model for molecular standardization.
We enable custom schemes based solely on data, which also support standardization options that are difﬁcult to encode into rules.
Our model achieves > 98% accuracy in learning two popular rule-based protocols.
When ﬁne-tuned on a relatively small dataset of catalysts (for which there is currently no automated standardization practice), the model predicts the expected standardized molecular format with a test accuracy of 62% on average.
We show that our model learns not only the grammar and syntax of molecular representations, but also the details of atom ordering, types of bonds, and representations of charged species.
In addition, we demonstrate the model’s ability to reproduce a canonicalization algorithm with a 95.6% success rate.
Introduction
From deep learning algorithms for forward reaction prediction [ 2, 3] and retrosynthesis [1, 4, 5], to the prediction of yields [6] and molecular properties [7], artiﬁcial intelligence has become an integral part of chemical discovery pipelines.
This was made possible thanks to the abundance of freely available molecular databases with hundreds of millions of compounds relevant to drug and materials discovery [8, 9, 10].
However, the size of the datasets makes human curation campaigns impractical, resulting in the frequent presence of incorrect and inconsistent molecular structure representations [11].
Because the quality of the input data limits the performance of machine learning models, the development of tools to address this issue has received increased attention in recent years [12].
In fact, a 2010 study [11] compiled a series of investigations which concluded that even minor structural errors and inconsistencies within a dataset could result in signiﬁcant losses in the predictive ability of structure-activity relationship models.
Standardization tools aim to correct errors in chemical structure representation, while also generating uniform and self-consistent conﬁgurations of atoms and bonds, charges and bond orders, aromaticity and stereochemistry.
[Figure] Currently, the only approach to chemical data standardization is to format compounds according to a set of rules and conventions [13, 14, 15].
These assume the occurrence of speciﬁc patterns in the arrangements of elements, bonds, and charges and necessitate the development of algorithms to convert them to a standard format (which often varies across organizations).
Manually crafted and coded rules have inherent disadvantages, the most notable of which is the need for programming expertise and time resources.
Even more importantly, it is not always possible to develop a set of rules to automate a chosen standardization protocol, even when experts design speciﬁc guidelines to manually standardize the corresponding compounds.
Figure 1 depicts a few examples of standardization transformations that cannot be mimicked by a human-written algorithm and thus require manual annotation as of today.
In this work, we propose a deep learning method based on the Transformer architecture [16] that converts molecules represented using the simpliﬁed molecular-input line-entry system (SMILES) [17, 18] from their non-standardized to their standardized format.
We demonstrate the versatility of the model by training it with two different standardization protocols and allowing the user to select the preferred protocol when standardizing new molecules.
We also leverage the pre-trained model to ﬁne-tune it on a dataset where the standardization process cannot be reduced to a set of rules.
Thus, when ﬁne-tuned on a dataset consisting of few hundreds of molecules that have been standardized based on human annotation, our model can capture commonalities in molecular structure representation and codify a speciﬁc set of rules for consistently modifying compounds.
Method 2.1 Model
We repurposed the transformer architecture [16] to execute a translation task from non-standardized to standardized molecular representations.
Tokenized SMILES strings are used as source and target inputs to the model and tokenization is performed using a custom regular expression pattern (see
Appendix A).
Among other aspects, the tokenization ensures the separation of metal atoms and their charge, which simpliﬁes the learning of atom identity rather than its charged state.
The transformer model was implemented using the OpenNMT-py library (version 1.0.0).
The parameters used are reported in Appendix B.
2.2 Data
The model was trained and tested on compounds belonging to PubChem [8], an open archive of chemical compounds.
The data (220k molecules) was acquired in non-standardized and in standardized formats which served as source and target inputs, respectively.
We performed validation on 8k compounds and tested on 12k compounds.
Upon inspection of the PubChem standardization protocol, we observed that several compounds are subject to addition of stereochemistry based on provided 3D information for the molecule.
In absence of 3D information as input to our model, we removed stereochemistry when comparing predictions to true values.
To evaluate the model on another standardization scheme, the source molecules extracted from
PubChem were standardized using the ChEMBL curation protocol [14].
Finally, a private dataset is introduced in this work and comprises 866 catalyst molecules which we standardized manually.
This was further split into training (766 molecules), validation (50 molecules) and test (50 molecules) sets.
We performed 5 splits and report average performance.
Results and discussion 3.1 Standardization learning
The following is an analysis of the ability of the model to perform various types of molecule standardization.
We used three different datasets to train the model.
The source molecules are ﬁxed across them and the targets are generated using: 1) the ChEMBL standardization rules, 2)
the
PubChem standardization rules, and 3) a combination of the two procedures.
[Table]
First, we show that the model can be adapted to learn different standardization protocols, to accommodate distinct preferences in molecular formatting.
Table 1 contains a summary of the results, including accuracy on the entire test dataset as well as solely on the compounds modiﬁed during rule-based standardization.
The datasets were split in two ways: randomly and based on Tanimoto indexes, which are a popular measure of the structural similarity between compounds [19].
As such, we adopted the method of Kovács et al.
[20] to allocate compounds to the training/test datasets so that no compound in the test set is within Tanimoto similarity σ = 0 .
6 of any compound in the training set.
The intent of such a split is to avoid structural bias and to make a robust evaluation of the model’s ability to generalise to unseen structures.
ChEMBL The results from the random split reveal that the model successfully learns the task, with an accuracy of 94.5% for compounds standardized using the ChEMBL protocol [14].
This involves modiﬁcations such as converting covalently drawn alkaline metals connected to O or N to ionic forms (e.g. NaO to Na + O − ), standardizing diazonium N to N + , removing explicit H atoms etc.
The model recognizes the molecules that can be standardized and achieves an overall test accuracy of 98.8%.
Hence, the model can also predict that the compounds do not require modiﬁcation and leaves the string unchanged.
When using a Tanimoto split, the accuracy drops to 87.8%, which is a testament to the scaffold bias introduced by the nature of the dataset.
PubChem Contrary to ChEMBL, PubChem standardization [13] uses a more extensive list of rules relying on routines from the OpenEye C++ toolkits [21].
Figure 2 exempliﬁes a few distinctions between the two approaches.
While the ChEMBL protocol maintains the orginal stereochemistry of
ChEMBL
Catalysts
PubChem
D EE P
S T A N
D A R D
I Z A T
I
O N
M O D
E L S
ChEMBL
Catalysts
PubChem
D EE P
S T A N
D A R D
I Z A T
I
O N
M O D
E L S
Invalid smiles the double bond, PubChem standardization converts the original geometry to an unspeciﬁed conﬁguration (represented by the wiggly bond). PubChem also standardizes the amine to its protonated state. Overall, with a random split, the model learns the PubChem protocol with a test set accuracy of 98.0%. For compounds that require modiﬁcation, 91.5% of the predictions match the real structure.
ChEMBL and PubChem
Finally, we explored a model that combines the two different standardization protocols presented previously in a prompt-based fashion.
The assumption is that competing probability distributions, modelling different standardization protocols, can mutually learn from each other, delivering substantial beneﬁt to the learning.
In practice, a [CHEMBL] /
[PUBCHEM] token was added to the input SMILES sequence to designate the preferred type of standardization.
The two protocols were successfully learned in a combined model, with an overall test accuracy of 92.7%.
We determined the test accuracies with which speciﬁcally the ChEMBL / PubChem rules were learned in the combined model.
These remain unaltered compared to when the model was trained on one protocol only: 94.3% for ChEMBL and 91.4% for PubChem, which is in line with the expected cross-task beneﬁts of multitasking [22].
This is a key ﬁnding that enables the user to train on multiple standardization protocols and query the model effectively.
3.2 Model transferability
Transferability was assessed by ﬁne-tuning the above trained models (see Table 1) on a private human-curated catalyst dataset.
Not only do the compounds in this set deviate from the pre-training data in terms of class and vocabulary, but the standardization rules used here are also unique.
The performance showed a maximum test set accuracy of 62 . 0 ± 4 .
0% (see Table 2) on a relatively small dataset (see Section 2.2), an increase of 18% compared to a non-pre-trained model.
The learning abilities of the model for this dataset are notable, as the performance of a null model in this case (which always returns the input string as a prediction) is 0%.
It is also worth noting the variety and complexities of the transformations.
Particularly important for catalysts, the model appears to learn the ligation preferences of metal centres.
Fig.
1 shows how distinct metals exhibit different coordination behaviours, which are otherwise challenging to capture with rule-based algorithms.
Upon visual inspection of wrong predictions, we observed that the model is prone to fail for large molecules, however the limited size of the test set does not allow for a proper statistical evaluation and further investigation is needed.
Fig.
3 reveals that incorrect predictions can be associated to [Table] [Figure] incomplete structures or incorrect standardization, and not to chemically invalid structures.
This ﬁnding, along with the boost in accuracy highlighted in Table 2, suggests that pre-training contributes to preventing the violation of chemical rules, while ﬁne-tuning is accountable for tailoring the model to a speciﬁc set of standardization rules.
At the same time, we notice that the ﬁne-tuned model loses its ability to standardize molecules belonging to the pre-training dataset (see Fig. 2).
A multitask learning approach could in principle overcome the issue.
3.3 Learning canonicalization
As a ﬁnal experiment, we randomized SMILES strings by doing a cyclic rotation of the atomic indices, and presented the model with the task of recovering the canonical analogues of the molecules.
This was achieved with an accuracy of 95.6% on a random test set.
Canonicalization was performed using the algorithm provided by RDKit [23].
Conclusions
The model for molecule standardization presented in this work is the ﬁrst attempt to replace its rule-based predecessors.
We have demonstrated a robust method that accommodates the variety of molecular representations that exist today.
First, it learns two popular standardization procedures with accuracies > 98% .
For the compounds that undergo modiﬁcations during rule-based protocols, the model predicts the correct outcome with test set accuracies > 91% .
The model can be trained on multiple procedures simultaneously, allowing the user to query it in a prompt-based fashion and select the preferred standardization practice.
Importantly, this does not reduce the standardization accuracy.
When presented with a small catalyst dataset with numerous formatting possibilities, the model learned the preferred standardizations with an average test accuracy of 62%.
Introducing additional features to the current development may further extend the model’s capabilities.
For example, we plan to explore the effect of the representation of source molecules on the predictive abilities of the model.
This will build towards the goal to learn an even more robust standardization, irrespective of the ﬂavour of the input SMILES.
A Tokenizer
SMILES strings were tokenized using the following regex expression: (\%\([0-9]{3}\)|\[[^\]]+]|Br?|Cl?|N|O|S|P|F|I|b|c|n|o|s|p|\||\(|\)|\.|=|#|-|\+|\\|\/ |:|~|@|\?|>>?|\*|\$|\%[0-9]{2}|[0-9])
The resulting tokenized string was post-processed using the following expression, to introduce a space between metal atoms and their [0-9]*)?\]-]charge: \[([A-Z][a-z]?)([+-][0-9]*)?\]
B Model implementation and training
The model is based on the default transformer implementation provided by OpenNMT-py , which was adapted through the following changes: the parameter layers was set to 4, rnn_size to 256, word_vec_size to 256, max_generator_batches