I nstruct P rotein: Aligning Human and Protein Language via Knowledge Instruction (2024)

Abstract: Large Language Models (LLMs) have revolutionized the field of natural language processing, but they fall short in comprehending biological sequences such as proteins. To address this challenge, we propose InstructProtein, an innovative LLM that possesses bidirectional generation capabilities in both human and protein languages: (i) taking a protein sequence as input to predict its textual function description and (ii) using natural language to prompt protein sequence generation. To achieve this, we first pre-train an LLM on both protein and natural language corpora, enabling it to comprehend individual languages. Then supervised instruction tuning is employed to facilitate the alignment of these two distinct languages. Herein, we introduce a knowledge graph-based instruction generation framework to construct a high-quality instruction dataset, addressing the annotation imbalance and the absence of instructional signals in the existing protein-text corpus. In particular, the instructions inherit the structural relations between proteins and function annotations in knowledge graphs, which empowers our model to engage in the causal modeling of protein functions, akin to the chain-of-thought processes in natural languages. Extensive experiments on bidirectional protein-text generation tasks show that InstructProtein outperforms state-of-the-art LLMs by a large margin.

## introduction the landscape of natural language processing (nlp) research, and indeed the broader artificial intelligence (ai) community, has recently been revolutionized by generative large language models (llms) @xcite @xcite , such as chatgpt (ouyang * corresponding author. @xcite . the expansion of parameter size and training corpora has empowered these models to acquire versatile, general-purpose data representations that seamlessly transcend linguistic tasks encompassing comprehension and generation in a multitude of languages. beyond natural languages (a.k.a., human languages), recent investigations have illuminated the potential of these llms to serve as a versatile interface for processing multimodal data, including but not limited to images, videos and speech @xcite @xcite . however, general llms fall short of capturing the intricate realm of biological sequences, a domain abundant with its own unique linguistic nuances. the biological sequences, particularly proteins, represent a distinctive facet of what could be referred to as "life language", exerting a significant influence on signal transduction pathways, enzymatic catalysis, and gene regulation @xcite @xcite @xcite . existing llms like chatgpt or gpt4 @xcite fail to accurately model the biological sequences, resulting in limitations on protein understanding and generation @xcite . to unlock the potential within llms for deciphering proteins, researchers have put rich efforts into developing protein language models (plms) @xcite @xcite @xcite . these specialized models are tailored to ingest amino acid sequences as inputs, predict protein functionalities, or even design de novo proteins. notwithstanding, it is crucial to highlight that while plms exhibit competence in comprehending amino acid sequences, they are unable to comprehend the intricacies of human languages. recent studies @xcite can accept both protein sequences and textual descriptions as input, aiming to enhance protein function prediction. nevertheless, these endeavors to align the realms of protein and human languages are unidirectional and remain in their nascent stages; they fall short of being able to generate protein sequences based on textual instructions. in essence, there exists an unaddressed void in the current landscape of llms, wherein the ability to swiftly traverse between human and protein languages. to enable an llm to adeptly comprehend both human and protein languages, we contend that the limitations imposed by existing models primarily stem from their training corpora. notably, many existing models are trained on either human languages or protein sequences, rendering them proficient in only one of these linguistic realms. this unilateral training approach is insufficient to imbue an llm with a comprehensive vocabulary encompassing both languages. moreover, it is important to recognize that the existing protein-text corpus used in previous studies @xcite @xcite has its limitations. (1) the imbalance of annotations: researchers tend to focus on well-studied proteins, leading to a significant disparity in the availability of annotations @xcite . training llms directly on such a corpus introduces model bias, which ultimately results in suboptimal performance. (2) the absence of instructional signals: protein-related textual content is primarily comprised of descriptive narratives, often devoid of instructional signals specifically designed for training llms. this inherent disparity obstructs a holistic understanding of a wide range of tasks, ultimately resulting in subpar zero-shot performance @xcite . in short, the fundamental hurdle of current llms involves curating an elaborate training corpus that seamlessly bridges the gap between human and protein languages. in this work, we introduce instructprotein, a pioneering study that aligns human and protein languages through knowledge instruction, enabling an llm with bidirectional generation capabilities between these two languages. specifically, to equip llms with the ability to understand protein language, instructprotein adopts a two-step training approach. it initiates with pre-training on protein and natural language corpora, followed by finetuning with the established protein knowledge instruction dataset. to construct such an instruction dataset, we first transform raw protein-text corpora into a structured knowledge graph @xcite . inspired by the idea of chain-of-thoughts, we enrich kg our contributions can be summarized as follows: ## a closer look at annotation imbalance much of life science research is dedicated to unraveling the biological functions of proteins. while certain proteins have undergone extensive investigation, there still exist tens of thousands of proteins remaining categorized as understudied. this phenomenon implies an imbalance in protein function annotation. to clearly illustrate this problem, we take the subcellular location as an example, and show its annotation distribution in figure 1 . the ## instructprotein this section presents the method of instructprotein. we first pre-train it in a self-supervised manner on natural language and protein sequence datasets respectively, and then conduct supervised tuning using the created knowledge instruction dataset. ## conclusion instructprotein explores the feasibility of bidirectional generation between human and protein languages within a single large language model. our approach involved the transformation of a raw protein-text corpus into a structured knowledge graph, from which kg triples were sampled and converted into instructions. this kg-based instruction generation method resulted in a high-quality instruction dataset, facilitating the llm to align protein language with human language.