B io T 5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning (2024)

Abstract: Recent research trends in computational biology have increasingly focused on integrating text and bio-entity modeling, especially in the context of molecules and proteins. However, previous efforts like BioT5 faced challenges in generalizing across diverse tasks and lacked a nuanced understanding of molecular structures, particularly in their textual representations (e.g., IUPAC). This paper introduces BioT5+, an extension of the BioT5 framework, tailored to enhance biological research and drug discovery. BioT5+ incorporates several novel features: integration of IUPAC names for molecular understanding, inclusion of extensive bio-text and molecule data from sources like bioRxiv and PubChem, the multi-task instruction tuning for generality across tasks, and a numerical tokenization technique for improved processing of numerical data. These enhancements allow BioT5+ to bridge the gap between molecular representations and their textual descriptions, providing a more holistic understanding of biological entities, and largely improving the grounded reasoning of bio-text and bio-sequences. The model is pre-trained and fine-tuned with a large number of experiments, including 3 types of problems (classification, regression, generation), 15 kinds of tasks, and 21 total benchmark datasets , demonstrating the remarkable performance and state-of-the-art results in most cases. BioT5+ stands out for its ability to capture intricate relationships in biological data, thereby contributing significantly to bioinformatics and computational biology. Our code is available at https://github.com/QizhiPei/BioT5.

## introduction molecules and proteins are two crucial bio-entities in drug discovery, forming the foundation of biological activities @xcite . a molecule can be represented by its smiles @xcite or selfies @xcite sequence, and a protein can be described by a fasta @xcite sequence. with the advancement of language models (lms), an increasing body of work focuses on understanding the molecules and proteins by modeling their biosequences @xcite @xcite . notably, biological literature @xcite is full of extensive information on molecules and proteins. when a biological entity is mentioned in such literature, its context is predominantly centered around a description of some characteristics of the entity. consequently, there has been a growing body of work dedicated to the joint modeling of text and biological entities @xcite , such as galactica @xcite , molxpt @xcite , biot5 @xcite and biomedgpt @xcite , which are all scientific models trained on text, molecule and protein sequences. despite their achievements, substantial opportunities for enhancement still remain: (1) prior works neglect the importance of modeling the textual name of molecules, such as international union of pure and applied chemistry (iupac), which provides a standard and systematic naming method for ensuring uniformity and clarity across the scientific community. different from smiles and selfies, iupac bears a closer resemblance to natural language that is evident in its widespread adoption within scientific literature @xcite . (2) previous models were predominantly specialist models, necessitating the training of a separate model for each downstream task, thereby lacking in generality and increasing the training and developing cost @xcite . (3) most of the previous models based on t5 @xcite and gpt @xcite architectures only focus on the classification tasks since they do not implement specialized tokenization for numerical data, which results in their suboptimal adaptation to regression tasks. to address the above challenges, in this paper, we introduce biot5+, an advanced iteration of the biot5 framework @xcite , designed to augment biological research and drug discovery with enriched data integration, multi-task capabilities, and the ability to solve regression tasks. shortly speaking, biot5+ incorporates following significant enhancements: with our designed pre-training and multi-task instruction tuning, the effectiveness of biot5+ is verified on 3 types of problems (classification, generation, and regression), 15 different tasks, and 21 benchmark datasets, including molecule property prediction, retrosynthesis, molecule description generation, drug-target interaction, and so on. biot5+ has shown highly competitive results, achieving state-of-the-art performance in most of the tasks. this robust performance underscores the enhanced capability of biot5+ to capture and analyze the intricate relationships and properties inherent in biological data, marking a significant step forward in computational biology. ## biot5+ framework this section introduces biot5+ framework and an overview is shown in figure 1 . the tasks involved in pre-training are presented in figure 3 . ## conclusions biot5+, as an advanced iteration of the biot5 framework, represents a significant stride in computational biology and drug discovery. by integrating iupac names, expanding bio-text and molecular data sources, employing multi-task instruction tuning, and incorporating an advanced numerical tokenization technique, biot5+ has successfully bridged the gap between molecular representations and their textual descriptions. the enhanced understanding of molecular structures and its ability to process complex biological data have been demonstrated across a wide range of tasks, with biot5+ achieving state-of-the-art performance in most of them. this success highlights the potential of biot5+ as a versatile and powerful tool in understanding and analyzing biological entities.