Translation between Molecules and Natural Language (2022)

Abstract: We present MolT5 - a self-supervised learning framework for pretraining models on a vast amount of unlabeled natural language text and molecule strings. MolT5 allows for new, useful, and challenging analogs of traditional vision-language tasks, such as molecule captioning and text-based de novo molecule generation (altogether: translation between molecules and language), which we explore for the first time. Since MolT5 pretrains models on single-modal data, it helps overcome the chemistry domain shortcoming of data scarcity. Furthermore, we consider several metrics, including a new cross-modal embedding-based metric, to evaluate the tasks of molecule captioning and text-based molecule generation. Our results show that MolT5-based models are able to generate outputs, both molecules and captions, which in many cases are high quality.

## introduction imagine a future where a doctor can write a few sentences describing a specialized drug for treating a patient and then receive the exact structure of the desired drug. although this seems like science fiction now, with progress in integrating natural language and molecules, it might well be possible in the future. historically, drug creation has commonly been done by humans who design and build individual molecules. in fact, bringing a new drug to market can cost over a billion dollars and take over ten years @xcite . recently, there has been considerable interest in using new deep learning tools to facilitate in silico drug design-a field often called cheminformatics @xcite . yet, many of these experiments still focus on molecules and their low-level properties such as logp (the octanol-water partition coefficient) @xcite . in the future, * indicates equal contributions. 1 all resources are publicly available at github.com/blendernlp/molt5 the molecule is an eighteen-membered homodetic cyclic peptide which is isolated from oscillatoria sp. and exhibits antimalarial activity against the w2 chloroquine-resistant strain of the malarial parasite, plasmodium falciparum. it has a role as a metabolite and an antimalarial. it is a homodetic cyclic peptide, a member of 1,3oxazoles, a member of 1,3-thiazoles and a macrocycle. ## molt5 -multimodal text-molecule representation model we can crawl a massive amount of text from the internet. for example, @xcite built a common crawl-based dataset that contains over 700 gb of reasonably clean and natural english text. on the other hand, over a billion molecules are also available from public databases such as zinc-15 @xcite . inspired by the progress in large-scale pretraining @xcite , we propose a new self-supervised learning framework named molt5 (molecular t5) to leverage the vast amount of unlabeled natural language text and molecule strings. figure 3 shows an overview of molt5. we first initialize an encoder-decoder transformer model @xcite using one of the public checkpoints of t5.1.1 2 , an improved version of t5 (raf-2 https://tinyurl.com/t511-ckpts fel et al., 2020). after that, we pretrain the model using the "replace corrupted spans" objective @xcite . more specifically, during each pretraining step, we sample a minibatch comprising both natural language sequences and smiles sequences. for each sequence, some words in the sequence are randomly chosen for corruption. each consecutive span of corrupted tokens is replaced by a sentinel token (shown as [x] and [y] in figure 3 ). then the task is to predict the dropped-out spans. 3 molecules (e.g. represented as smiles strings) can be thought of as a language with a very unique grammar. then, intuitively, our pretraining stage essentially trains a single language model on two monolingual corpora from two different languages, and there is no explicit alignment between the two corpora. this approach is similar to how some multilingual language models such as mbert @xcite and mbart @xcite were pretrained. as models such as mbert demonstrate ex-cellent cross-lingual capabilities @xcite , we also expect models pretrained using molt5 to be useful for text-molecule translation tasks. after the pretraining process, we can finetune the pretrained model for either molecule captioning or generation (depicted by the bottom half of figure 3 ). in molecule generation, the input is a description, and the output is the smiles representation of the target molecule. on the other hand, in molecule captioning, the input is the smiles string of some molecule, and the output is a caption describing the input molecule. ## 2 the molecule is a hydrate that is the dihydrate form of manganese(ii) chloride. it has a role as a mri contrast agent and a nutraceutical. it is a hydrate, an inorganic chloride and a manganese coordination entity. ## broader impacts our proposed model and tasks will have the following broader impacts. 1) it will help to democratize molecular ai, allowing chemistry experts to take advantage of new ai technologies for discovering new life-changing drugs by interacting in the natural language, because it is most natural for humans to provide explanations and requirements in natural language. 2) text-based molecule generation enables the ability to generate molecules with specific functions (such as taste) rather than properties, enabling the next generation of chemistry where custom molecules are used for each application. specifically-designed molecular solutions have the potential to revolutionize fields such as medicine and material science. 3) our models, whose weights we will release, will allow further research in the nlp community on the applications of multimodal text-molecule models. ## limitations since this work focuses on a new application for large language models, many of the same limitations apply here. namely, the model is trained on a large dataset collected from the internet, so it may contain unintended biases. one limitation of our model is using smiles strings -recent work @xcite proposes a string representation with validity guarantees. in practice, we found this to work poorly with pretrained t5 checkpoints (which were important from a computational perspective). we also note that some compounds in chebi-20 can cause validity problems in the default selfies implementation. we leave further investigation of this to future work. finally, we stress that molt5 was created for research purposes and generated molecules should not be used for medical purposes without careful evaluation by standard clinical testing first. ## 12 the molecule is a member of the class of chloroethanes that is ethane in which five of the six hydrogens are replaced by chlorines. a non-flammable, high-boiling liquid (b.p. 161-162℃) with relative density 1.67 and an odour resembling that of chloroform, it is used as a solvent for oil and grease, in metal cleaning, and in the separation of coal from impurities. it has a role as a non-polar solvent. ## 13 transformer rnn t5 molt5 input ground truth the molecule is an eighteen-membered homodetic cyclic peptide which is isolated from oscillatoria sp. and exhibits antimalarial activity against the w2 chloroquine-resistant strain of the malarial parasite, plasmodium falciparum. it has a role as a metabolite and an antimalarial. it is a homodetic cyclic peptide, a member of 1,3-oxazoles, a member of 1,3-thiazoles and a macrocycle. the molecule is an n-carbamoylamino acid that is aspartic acid with one of its amino hydrogens replaced by a carbamoyl group. it has a role as a saccharomyces cerevisiae metabolite, an escherichia coli metabolite and a human metabolite. it is a n-carbamoyl-amino acid, an aspartic acid derivative and a c4dicarboxylic acid. it is a conjugate acid of a n-carbamoylaspartate(2-). the molecule is a tripeptide composed of glycine, glycine and l-alanine residues joined in sequence. it has a role as a metabolite. figure 7 : more examples of interesting molecules generated by different models. the molecule is a methylindole carrying a methyl substituent at position 3. it is produced during the anoxic metabolism of l-tryptophan in the mammalian digestive tract. it has a role as a mammalian metabolite and a human metabolite. the molecule is a member of the class of xanthenes that is used as a zn(2+)selective fluorescent indicator. it has a role as a histological dye, a chelator and a visual indicator. it is a member of xanthenes, a cyclic ketone, an aromatic ether, a member of phenols, an organofluorine compound, a tricarboxylic acid and a substituted aniline. the molecule is an acyl-coa that results from the formal condensation of the thiol group of coenzyme a with the carboxy group of (e)-2-benzylidenesuccinic acid. it is a conjugate acid of an (e)-2benzylidenesuccinyl-coa(5-). ## 5 6 the molecule is a dtdpsugar having 4 -dehydro -6, 6 -dideoxy -alpha -d -manno -oct -2ulosonic acid. it has a role as an escherichia coli metabolite and a mouse metabolite. it is a conjugate acid of a dtdpalpha -d -glucose ( 2 -). the molecule is the stable isotope of helium with relative atomic mass 3. 016029. the least abundant ( 0. 000137 atom percent ) isotope of naturally occurring helium. the molecule is a dtdpsugar having 4-dehydro-2,6-dideoxy-beta-lglucose as the sugar component. it is a dtdpsugar and a secondary alpha-hydroxy ketone. it derives from a dtdp-lglucose. the molecule is a dtdpsugar having 4-dehydro-2,6-dideoxy-alpha-dglucose as the sugar component. it is a dtdpsugar and a secondary alpha-hydroxy ketone. it derives from a dtdp-dglucose. the molecule is a dtdp-sugar having 4-dehydro-2,6-dideoxyalpha-d-glucose as the sugar component. it has a role as a bacterial metabolite. it is a dtdpsugar and a secondary alphahydroxy ketone. it derives from a dtdp-d-glucose. it is a conjugate acid of a dtdp-4-dehydro-2,6dideoxy-alpha-d-glucose(2-). the molecule is a tetrapeptide composed of l -asparagine, l -aspartyl, l -aspartic acid, and laspartic acid units joined in sequence by peptide linkages. it has a role as a metabolite. it derives from a l -glutamic acid. the molecule is the stable isotope of oxygen with relative atomic mass 15. 99. the most abundant ( 99. 99 atom percent ) isotope of naturally occurring oxygen. the molecule is a tripeptide composed of two l-leucine units joined to l-aspartic acid by a peptide linkage. it has a role as a metabolite. it derives from a lleucine and a l-aspartic acid. the molecule is a tripeptide composed of l-leucine, lvaline and l-aspartic acid joined in sequence by peptide linkages. it has a role as a metabolite. it derives from a l-leucine, a l-valine and a l-aspartic acid. the molecule is a tripeptide composed of l-leucine, lvaline and l-aspartic acid joined in sequence by peptide linkages. it has a role as a metabolite. it derives from a l-leucine, a l-valine and a l-aspartic acid. figure 9 : more examples of interesting captions generated by different models. 400 figure 25: input: the molecule is a orange dye. 402 figure 31: input: the molecule is a solid at room temperature. 403 figure 34: input: the molecule is a topical anaesthetic. 404 figure 37: input: the molecule is an agrochemical. 405 figure 40: input: the molecule is an angry man. 406 figure 43: input: the molecule is an anti-inflammatory agent. 407 figure 46: input: the molecule is an antipruritic drug. 408 figure 49: input: the molecule is an aromatic ether. 409 figure 52: input: the molecule is an inhibitor of the parkinson's disease. 411 figure 57: input: the molecule is blue blue. 412 figure 60: input: the molecule is blue blue blue blue blue.