Multilingual Molecular Representation Learning via Contrastive Pre-training (2022)

Abstract: Molecular representation learning plays an essential role in cheminformatics. Recently, language model-based approaches have gained popularity as an alternative to traditional expert-designed features to encode molecules. However, these approaches only utilize a single molecular language for representation learning. Motivated by the fact that a given molecule can be described using different languages such as Simplified Molecular Line Entry System (SMILES), The International Union of Pure and Applied Chemistry (IUPAC), and The IUPAC International Chemical Identifier (InChI), we propose a multilingual molecular embedding generation approach called MM-Deacon (multilingual molecular domain embedding analysis via contrastive learning). MM-Deacon is pre-trained using SMILES and IUPAC as two different languages on large-scale molecules. We evaluated the robustness of our method on seven molecular property prediction tasks from MoleculeNet benchmark, zero-shot cross-lingual retrieval, and a drug-drug interaction prediction task.

## introduction drug discovery process involves screening of millions of compounds in the early stages of drug design, which is time consuming and expensive. computer-aided drug discovery can reduce the time and cost involved in this process via automating various cheminformatics tasks @xcite @xcite . traditional methods to encode molecules such as fingerprint generation rely heavily on molecular fragment-level operations on top of molecule graph constructed by molecular atoms and bonds @xcite @xcite . an example of such methods is morgan fingerprint, also known as extended-connectivity fingerprint (ecfp) @xcite , where a fixed binary hash function is applied on each atom and its neighborhood. these kinds of approaches focus on local features, hence they may not capture global information. in addition to molecule graph, a given molecule can also be described with different languages such as simplified molecular line entry system (smiles), the international union of pure and applied chemistry (iupac), and the iupac international chemical identifier (inchi). particularly, smiles is widely used to represent molecule structures as ascii strings @xcite at an atom and bond level. iupac nomenclature, on the other hand, serves the purpose of systematically naming organic compounds by basic words that indicate the structure of the compound and prioritize on functional groups to facilitate communication @xcite . fig. 1 shows a comparison of smiles and iupac characteristics for the same molecule. the smiles string is created by traversing the molecule graph, where each letter in the smiles string (such as c, f, n, o in fig. 1 ) corresponds to an atom on the graph, and other characters represent positions and connectivity. however, iupac names are akin to a natural language, and morphemes in the iu-pac name (like fluoro, prop, en, yl in this example) often represent specific types of substructure on the molecule graph, which are also responsible for characteristic chemical reactions of molecules. advances in natural language processing (nlp) have been very promising for molecule embedding generation and molecular property prediction @xcite @xcite @xcite @xcite @xcite . it is important to note that all of the methods mentioned above work with smiles representation only. therefore, the underlying chemical knowledge encoded in the em-figure 1 : schematic illustration of differences in smiles and iupac characteristics for the same molecule. 2d molecule graphs are plotted using rd-kit @xcite . for the sake of simplicity, we only highlight atom c in both smiles string and the molecule graph below it. for iupac, we use the same color to denote the correspondence between the textual description in the string and the substructure on the molecule graph, where orange represents fluorosubstituent, yellow represents benz-ene, light green represents nitro-, and turquoise represents prop-en-yl. in recent years, contrastive learning has been successful in multimodal vision and language research @xcite @xcite @xcite @xcite @xcite . @xcite used image-text pairs to learn scalable visual representations. @xcite showed the superiority of contrastive objectives in acquiring global (not fragment-level) semantic representations. in light of these advances, we propose mm-deacon (multilingual molecular domain embedding analysis via contrastive learning), a molecular representation learning algorithm built on smiles and iupac joint training. transformers are used as base encoders in mm-deacon to encode smiles and iupac, and embeddings from encoders are projected to a joint embedding space. afterwards, a contrastive objective is used to push the embeddings of positive cross-lingual pairs (smiles and iupac for the same molecule) closer together and the embeddings of negative cross-lingual pairs (smiles and iupac for different molecules) farther apart. here instead of using smiles and iupac for sequence-to-sequence translation @xcite @xcite , we obtain positive and negative smiles-iupac pairs and contrast their embeddings at the global molecule level rather than the fragment level. different molecule descriptors are thus integrated into the same joint embedding space, with mutual information maximized across distinct molecule languages. we pre-train mm-deacon on 10 million molecules chosen at random from the publicly available pubchem dataset @xcite and then use the pre-trained model for downstream tasks. our main contributions are as follows: deep learning tasks commonly face two challenges: first, dataset size is often limited, and second, annotations are scarce and expensive. a pre-training scheme can benefit downstream tasks by leveraging large-scale unlabeled or weakly labeled data. such pre-training and fine-tuning frameworks have recently sparked much interest in the molecular domain @xcite @xcite @xcite @xcite @xcite . existing pre-training methods can be divided into three categories based on the models used: pre-training with graph neural networks (gnns), pre-training with language models, and pre-training with hybrid models. pre-training with gnns. gnns are a popular choice for molecule encoding that regard atoms as nodes and bonds as edges. @xcite pretrained gnns on 2 million molecules using both node-level and graph-level representations with attribute masking and structure prediction objectives. molclr @xcite used subgraph-level molecule data augmentation scheme to create positive and negative pairs and contrastive learning to distinguish positive from negative. graphmvp @xcite was pre-trained on the consistency of 2d and 3d molecule graphs (3d graphs formed by adding atom spatial positions to 2d graphs) and contrastive objectives with gnns. pre-training with language models. language models are widely used to encode smiles for molecular representation learning. xu et al. (2017) reconstructed smiles using encoder-decoder gated recurrent units (grus) with seq2seq loss, where embeddings in the latent space were used for downstream molecular property prediction. chemberta (chithrananda et al., 2020) fed smiles into transformers, which were then optimized by mlm loss. fragnet (shrivastava and kell, 2021) used encoder-decoder transformers to reconstruct smiles and enforced extra supervision to the latent space with augmented smiles and contrastive learning. x-mol (xue et al., 2021) was pretrained by taking as input a pair of smiles variants for the same molecule and generating one of the two input smiles as output with transformers on 1.1 billion molecules. pre-training with hybrid models. different molecule data formats can be used collaboratively to enforce cross-modality alignment, resulting in the use of hybrid models. for example, dmp (zhu et al., 2021) was built on the consistency of smiles and 2d molecule graphs, with smiles encoded by transformers and 2d molecule graphs encoded by gnns. unlike other molecule pre-training methods, mm-deacon is multilingually pre-trained with language models using pairwise smiles and iupac. compared with using molecule graphs with gnns, iupac names encoded by language models bring in a rich amount of prior knowledge by basic words representing functional groups, without the need for sophisticated graph hyperparameter design. ## experiments mm-deacon was evaluated on seven molecular property prediction tasks from moleculenet bench-mark @xcite , zero-shot cross-lingual retrieval, and a drug-drug interaction (ddi) prediction task. ## discussions after being pre-trained on 10 million molecules, mm-deacon showed outstanding knowledge transfer capabilities to various downstream scenarios (fig. 3 ) where a pre-trained model could be used. the competitive performance on seven molecular property prediction tasks from moleculenet benchmark demonstrated the effectiveness of the pre-trained mm-deacon when adopting a network fine-tuning scheme as shown in fig. 3(a ). the evaluation results of zero-shot cross-lingual retrieval further revealed that mm-deacon smiles and iu-pac fingerprints shared a substantial amount of mutual information, implying that an iupac name can be used directly without first being translated to smiles format as chemists have done in the past. the ddi prediction task showed that mm-deacon also allows directly using embeddings in the joint cross-modal space as molecular fingerprints for downstream prediction tasks, which is a widely used strategy in cheminformatics. mm-deacon profited from the alignment of two molecule languages with distinct forms of nomenclatures, as opposed to the baseline mlm-[cls] model, which was pre-trained on smiles representation only. furthermore, we looked at moleculelevel and token-level alignments of mm-deacon to untangle the outcome of cross-lingual contrastive learning. ## conclusion in this study, we proposed a novel method for multilingual molecular representation learning that combines mutual information from smiles-iupac joint training with a self-supervised contrastive loss. we evaluated our approach for molecular property prediction, zero-shot cross-lingual retrieval, and ddi prediction. our results demonstrate that the self-supervised multilingual contrastive learning framework holds enormous possibilities for chemical domain exploration and drug discovery. in future work, we plan to scale mm-deacon pretraining to larger dataset sizes, as well as investigate the applicability of mm-deacon to other types of molecule languages.