P rot T 3: Protein-to-Text Generation for Text-based Protein Understanding (2024)

Abstract: Language Models (LMs) excel in understanding textual descriptions of proteins, as evident in biomedical question-answering tasks. However, their capability falters with raw protein data, such as amino acid sequences, due to a deficit in pretraining on such data. Conversely, Protein Language Models (PLMs) can understand and convert protein data into high-quality representations, but struggle to process texts. To address their limitations, we introduce ProtT3, a framework for Protein-to-Text Generation for Text-based Protein Understanding. ProtT3 empowers an LM to understand protein sequences of amino acids by incorporating a PLM as its protein understanding module, enabling effective protein-to-text generation. This collaboration between PLM and LM is facilitated by a cross-modal projector (i.e., Q-Former) that bridges the modality gap between the PLM’s representation space and the LM’s input space. Unlike previous studies focusing on protein property prediction and protein-text retrieval, we delve into the largely unexplored field of protein-to-text generation. To facilitate comprehensive benchmarks and promote future research, we establish quantitative evaluations for protein-text modeling tasks, including protein captioning, protein question-answering, and protein-text retrieval. Our experiments show that ProtT3 substantially surpasses current baselines, with ablation studies further highlighting the efficacy of its core components. Our code is available at https://github.com/acharkq/ProtT3.

## introduction language models (lms) have achieved impressive successes across diverse domains @xcite @xcite . remarkably, the extensive biological literature in their ## related works here we briefly review relevant fields of plms, protein-text modeling, and multi-modal lms. protein language models (plms). plms are transformer-based lms pretrained on large corpora of protein sequences for protein understanding and protein generation @xcite @xcite @xcite @xcite . similar to lms for texts, plms are pretrained by the objective of masked language modeling @xcite or auto-regressive modeling @xcite . plms have demonstrated promising performances on downstream tasks of 3d structure prediction and protein property prediction @xcite . however, they cannot process texts due to the absence of text in their pretraining data. protein-text modeling. the pioneering study, galactica @xcite , is an lm pretrained on a text corpus that includes a small amount of protein sequences. its protein understanding ability can be suboptimal compared to plms due to insufficient protein pretraining data. to mitigate this gap, subsequent works @xcite jointly leverage a plm and an lm by cross-modal contrastive learning @xcite . however, cross-modal contrastive learning is insufficient for protein-to-text generation. this is because protein-to-text generation is a conditional generation task. it demands the lm to understand proteins as the generation condition, which contrastive learning cannot achieve. further, protein-chat @xcite attempts to enable an lm to understand proteins by training a linear projector between a plm and an lm. considering both the plm and lm are kept frozen during training, this approach struggles to capture the nonlinear re-lations between proteins and texts. in a different approach, ontoprotein @xcite finetunes a plm using a biology knowledge graph, focusing more on protein property prediction and less on text-related tasks. multi-modal lms. enabling lms to understand another modality, such as image @xcite , video @xcite , audio @xcite , user-items @xcite , and molecule @xcite @xcite @xcite , has been an activate research direction. notably, the research of multi-modal lms is pioneered by vision-language modeling (vlm). vlm has been successfully applied for few-shot image classification, image captioning, and image qa @xcite . specifically, to enable lms to understand images, the leading vlm methods employ either 1) nonlinear and expressive cross-modal projectors @xcite , or 2) fine-tuned visual encoders and lms on multi-modal datasets @xcite , while such studies are missing for protein-text modeling. our work takes the initiative to explore this direction. ## model architecture here we introduce prott3's three key components: a plm for protein understanding, an lm for text processing, and a cross-modal projector to bridge the modality gap between the first two components. protein language model (plm). we utilize esm-2 @xcite to encode protein sequences of amino acids. esm-2 is an encoder-only transformer lm @xcite pretrained on 60m protein sequences by masked language modeling @xcite . plms have shown promising performances for protein folding @xcite , multiple sequence alignment @xcite , and protein property prediction @xcite , demonstrating their effectiveness for capturing protein characteristics. for efficiency, we freeze esm-2's weights in our training process. language model (lm). we choose galactica @xcite as the base lm. galactica is a decoder-only transformer lm pretrained on a large collection of scientific papers, spanning disciplines like biology and medicine. notably, galactica has demonstrated a high-level understanding of protein concepts through its promising performances in biomedical qa benchmarks @xcite . furthermore, galac- self-attention cross-attention feed-forward msaksnkig vvqltilt… self-attention feed-forward catalyze both the uptake … protein-text matching protein captioning protein-text contrasting protein language model xn xn (a) cross-modal projector: q-former query input text input protein input text transformer protein transformer self-attention module bi-directional attention mask bi-directional attention mask protein-text contrasting protein-text matching protein captioning text input (b) self-attention module for different tasks query input bi-directional attention mask bi-directional attention mask causal attention mask tica includes a small set of protein sequences in its pretraining data, and shows a capability for understanding protein sequences through the task of protein keyword prediction @xcite . therefore, we also leverage galactica as a baseline for protein-to-text generation to ablate the effectiveness of incorporating an additional plm for protein understanding. cross-modal projector. we employ a crossmodal projector based on q-former @xcite to bridge the modality gap between the plm and the lm. q-former has demonstrated promising performances in vision-language tasks. as figure 3a illustrates, q-former consists of two transformers: one for protein encoding and another for text processing. specifically, the protein transformer maintains n q learnable query tokens {q @xmath0. these query tokens can interact with the plm through the crossattention modules, in order to extract protein features. we denote the protein transformer's output as @xmath1, containing protein features. for text input, the text transformer adds a [cls] token at the beginning, and uses the [cls] token's output as the text representation. both transformers share the self-attention to enable interactions between proteins and texts. details are in section 4.1. q-former's weights are initialized from pubmedbert-abstract @xcite , a bert lm pretrained on paper abstracts from the pubmed database 1 . it has shown promising performances in understanding biomedical concepts under the blurb benchmark @xcite . the cross-attention module is added into the q-former every two layers and is randomly initialized. ## training method in this section, we introduce prott3's two training stages: protein-text retrieval and protein-to-text 1 https://pubmed.ncbi.nlm.nih.gov/ generation. the training process leverages a dataset of protein-text pairs @xmath2, t 1 ), (p 2 , t 2 ), ...}, where p i is a protein sequence and t i is the corresponding text sequence. ## experiments we begin by introducing the datasets of protein-text pairs, followed by the experimental results. details on experimental settings, such as hyperparameters and baseline implementations, are provided in appendix c. in the experiments, prott3 utilizes galactica 1.3b as the base lm and esm-2 150m as the plm. ## conclusion and future works in this work, we propose prott3, a new proteintext modeling framework. prott3 aims to facilitate text-based protein understanding via protein-to-text generation and protein-text retrieval. to achieve this, prott3 integrates a plm into an lm to enhance the lm's protein understanding ability. a cross-modal projector enables this integration by bridging the modality gap between the two modules. to promote future research, we set benchmarks for protein-text modeling tasks, including protein captioning, protein qa, and protein-text retrieval, where prott3 significantly outperforms existing baselines. looking ahead, we plan to explore enabling lms to understand 3d protein structures @xcite and apply this understanding to more tasks of drug discovery, property prediction @xcite @xcite , molecule generation @xcite , and ood generalization @xcite .