B io T 5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations (2023)

Abstract: Recent advancements in biological research leverage the integration of molecules, proteins, and natural language to enhance drug discovery. However, current models exhibit several limitations, such as the generation of invalid molecular SMILES, underutilization of contextual information, and equal treatment of structured and unstructured knowledge. To address these issues, we propose BioT5, a comprehensive pre-training framework that enriches cross-modal integration in biology with chemical knowledge and natural language associations. BioT5 utilizes SELFIES for 100% robust molecular representations and extracts knowledge from the surrounding context of bio-entities in unstructured biological literature. Furthermore, BioT5 distinguishes between structured and unstructured knowledge, leading to more effective utilization of information. After fine-tuning, BioT5 shows superior performance across a wide range of tasks, demonstrating its strong capability of capturing underlying relations and properties of bio-entities. Our code is available at https://github.com/QizhiPei/BioT5.

## introduction molecules and proteins are two essential bioentities in drug discovery @xcite . small molecule drugs have been the cornerstone of the pharmaceutical industry for nearly a century, owing to their unique advantages such as oral availability, diverse modes of action, etc @xcite . proteins serve as the foundation of life science, functioning as drug targets or crucial elements in disease pathways. as illustrated in figure 1 , both molecule can be represented by its name, bio-sequence (smiles and selfies), and 2d graph structure. protein can be represented by its name, corresponding gene name, bio-sequence (fasta), and 3d structure. to address the above challenges, in this paper, we introduce biot5, a comprehensive pre-training framework encompassing text, molecules, and proteins. biot5 leverages selfies @xcite to represent small molecules since its advantage over smiles is that selfies offers a more robust and error-tolerant molecular representation, eliminating issues of illegitimate structures often encountered with smiles. there are mainly two steps for biot5 pre-training: after pre-training, we fine-tune the obtained biot5 on 15 tasks covering molecule and protein property prediction, drug-target interaction prediction, protein-protein interaction prediction, molecule captioning, and text-based molecule generation. biot5 achieves state-of-the-art performances on 10 tasks and exhibits results comparable to domain-specific large models on 5 tasks, demonstrating the superior ability of our proposed method. biot5 model establishes a promising avenue for the integration of chemical knowledge and natural language associations to augment the current understanding of biological systems. ## related work in this section, we briefly review related work about cross-modal models in biology and representations of molecule and protein. ## biot5 the overview of the biot5 pre-training is illustrated in figure 2 . we combine data from different modalities to perform multi-task pre-training. ## experiments and results we evaluate biot5 on 15 well-established downstream tasks, which can be categorized into three types: single-instance prediction, multi-instance prediction, and cross-modal generation. we include details regarding fine-tuning datasets, baselines, and prompts in appendix f. for the downstream binary classification tasks presented in section 4.1 and 4.2, the calculation of evaluation metrics such as auroc and auprc necessitates the soft probability of the predicted label. as we use the prompt-based fine-tuning method, the output is either yes for the positive label or no for the negative label. to obtain an appropriate label distribution, following @xcite , we first extract the probabilities of yes and no tokens (denoted as p pos and p neg respectively) and normalize them. the resulting probability for positive label is ppos ppos+pneg and negative label is pneg ppos+pneg . ## conclusions and future work in this paper, we propose biot5, a comprehensive pre-training framework capable of capturing the underlying relations and properties of bio-entities by leveraging both structured and unstructured data sources with 100% robust molecular representation. our method effectively enriches cross-modal 2 https://openai.com/blog/openai-api integration in biology with chemical knowledge and natural language associations, demonstrating notable improvements in various tasks. for future work, we aim to further enrich our model by incorporating additional biological data types, such as genomics or transcriptomics data, to create a more holistic biological pre-training framework. additionally, we plan to evaluate the interpretability of biot5 predictions, aiming to provide more insights into the biological systems under study. thus, we foresee our work sparking further innovation in the use of ai models in the field of computational biology, ultimately leading to a deeper understanding of biological systems and facilitating more efficient drug discovery.