P rot LLM : An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training (2024)

Abstract: We propose ProtLLM, a versatile cross-modal large language model (LLM) for both protein-centric and protein-language tasks. ProtLLM features a unique dynamic protein mounting mechanism, enabling it to handle complex inputs where the natural language text is interspersed with an arbitrary number of proteins. Besides, we propose the protein-as-word language modeling approach to train ProtLLM. By developing a specialized protein vocabulary, we equip the model with the capability to predict not just natural language but also proteins from a vast pool of candidates. Additionally, we construct a large-scale interleaved protein-text dataset, named InterPT, for pre-training. This dataset comprehensively encompasses both (1) structured data sources like protein annotations and (2) unstructured data sources like biological research papers, thereby endowing ProtLLM with crucial knowledge for understanding proteins. We evaluate ProtLLM on classic supervised protein-centric tasks and explore its novel protein-language applications. Experimental results demonstrate that ProtLLM not only achieves superior performance against protein-specialized baselines on protein-centric tasks but also induces zero-shot and in-context learning capabilities on protein-language tasks.

## introduction understanding proteins is essential for unraveling the mysteries of life and enabling artificial intelligence systems to advance bioscience research @xcite . thanks to the development of deep learning techniques, neural network models encompass extensive protein-centric applications, such as protein-folding prediction @xcite , protein-protein interaction analysis @xcite , function prediction @xcite , etc. ## methods in this section, we elaborate on our proposed method, protllm, which is illustrated in figure 2. initially, we detail the model architecture in section 3.1. subsequently, the pre-training strategy is explained, introducing the concept of proteinas-word modeling, as outlined in section 3.2. we then present the uniquely constructed interleaved protein-text dataset, interpt, in section 3.3. lastly, we explore the application of protllm on a variety of tasks in section 3.4. ## experiments we evaluate protllm on three types of downstream tasks: (1) protein-centric tasks, which include supervised fine-tuning on conventional benchmarks for protein understanding; ## conclusion in this paper, we present protllm, a versatile llm designed to tackle both protein-centric and protein-language tasks. through dynamic protein mounting and protein-as-word modeling, protllm adeptly handles complex interleaved protein-text data, seamlessly unifying a wide array of protein tasks via a natural language interface. besides, we construct a large-scale protein-language pre-training dataset, called interpt, which encourages the model to learn from diverse data sources ranging from structured paired data to unstructured multi-protein scientific articles. extensive experiments demonstrate that protllm not only achieves competitive performance against specialized baselines across standard protein-centric benchmarks but also paves the way for exploring novel protein-language applications.