M ol XPT : Wrapping Molecules with Text for Generative Pre-training (2023)

Abstract: Generative pre-trained Transformer (GPT) has demonstrates its great success in natural language processing and related techniques have been adapted into molecular modeling. Considering that text is the most important record for scientific discovery, in this paper, we propose MolXPT, a unified language model of text and molecules pre-trained on SMILES (a sequence representation of molecules) wrapped by text. Briefly, we detect the molecule names in each sequence and replace them to the corresponding SMILES. In this way, the SMILES could leverage the information from surrounding text, and vice versa. The above wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem are all fed into a language model for pre-training. Experimental results demonstrate that MolXPT outperforms strong baselines of molecular property prediction on MoleculeNet, performs comparably to the best model in text-molecule translation while using less than half of its parameters, and enables zero-shot molecular generation without finetuning.

## introduction generative pre-trained transformer (gpt), like gpt-3 @xcite and @xcite , have obtained great success in natural language processing. they usually have billions of parameters and are trained on large corpus @xcite . by witnessing their great power, people start transferring language models to chemical @xcite and biological domains @xcite . for example, a small molecule (e.g., an oral drug) can be represented using simplified molecular-input lineentry system (smiles) @xcite , which is a sequence obtained by traversing the molecular graph using depth-first-search and several rules for branching, aromaticity, etc. after serializing molecules, people pre-train language models on smiles @xcite @xcite and obtain promising results for molecular generation. text is the most important record for molecular science and more generally, scientific discovery @xcite . it describes detailed properties of molecules, like how to synthesize the molecule @xcite , whether the molecule is toxic @xcite , etc. biogpt @xcite and pubmedgpt @xcite are two language models trained on biomedical literature. recently, a new trend is to jointly model smiles and scientific text so as to obtain shared representations across the two modalities. molt5 is a t5-like @xcite ) model, where several spans of the text/smiles are masked in the encoder and they should be reconstructed in the decoder. galactica @xcite is a gptlike @xcite ) model pre-trained on various types of inputs, like text, smiles, protein sequences, etc. although those models demonstrate progress in prediction and generation tasks, they do not explicitly leverage the relation between molecules and text. an intuition is that, in scientific literature, when a molecule name appears in a sentence, the surrounding context could be a description of the molecule. this should be useful information for joint training but is ignored in those models. to leverage such relations, in this work, we propose a novel molecule-text language model (molxpt), which is trained on "wrapped" sequences: given a sentence, we detect the molecular names with named entity recognition tools, and if any, replace them to the corresponding smiles and obtain the "wrapped" sequence between smiles and text. we pre-train a 24-layer molxpt (with 350m parameters) on 8m wrapped sequences, as well as 30m smiles from pubchem after pre-training, we finetune molxpt on moleculenet (a benchmark about molecular property prediction) @xcite and molecule-text translation @xcite using promptbased finetuning. on moleculenet, molxpt outperforms strong baselines with sophisticated design like gem @xcite . on text-molecule translation, molxpt performs comparably with the state-of-the-art model, molt5-large @xcite . molt5-large has 800m parameters while molxpt only uses 44% of its parameters. we also verify that molxpt has the zero-shot ability on text-to-molecule generation. ## our method molxpt is a language model pre-trained on heterogeneous data including scientific text, smiles sequences, and "wrapped" sequences between smiles and text. due to the flexible input, we can finetune it for various text and molecular tasks. the framework of molxpt is in figure 1 . ## experiments we evaluated molxpt on two downstream tasks: ## conclusions and future work we propose molxpt, a generative model pretrained on scientific text, molecular smiles and for future work, first, we will train larger molxpt to further verify the performances across different tasks and the zero-shot/in-context (xie et al., 2022) learning ability. second, how to further enhance the interaction between molecules and text (e.g., using contrastive learning to enhance consistency) should be studied. third, how to effectively adapt molxpt into other molecule and text tasks such as text-guided molecule optimization is another direction to explore.