Design Proteins Using Large Language Models: Enhancements and Comparative Analyses (2024)

Abstract: Pre-trained LLMs have demonstrated substantial capabilities across a range of conventional natural language processing (NLP) tasks, such as summarization and entity recognition. In this paper, we explore the application of LLMs in the generation of high-quality protein sequences. Specifically, we adopt a suite of pre-trained LLMs, including Mistral-7B, Llama-2-7B, Llama-3-8B, and gemma-7B, to produce valid protein sequences. All of these models are publicly available (https://github.com/KamyarZeinalipour/protein-design-LLMs).Unlike previous work in this field, our approach utilizes a relatively small dataset comprising 42,000 distinct human protein sequences. We retrain these models to process protein-related data, ensuring the generation of biologically feasible protein structures. Our findings demonstrate that even with limited data, the adapted models exhibit efficiency comparable to established protein-focused models such as ProGen varieties, ProtGPT2, and ProLLaMA, which were trained on millions of protein sequences. To validate and quantify the performance of our models, we conduct comparative analyses employing standard metrics such as pLDDT, RMSD, TM-score, and REU. Furthermore, we commit to making the trained versions of all four models publicly available, fostering greater transparency and collaboration in the field of computational biology.

## introduction in recent years, the field of natural language processing (nlp) has achieved remarkable progress, particularly through the development and utilization of large pre-trained language models. these sophisticated models represent a significant leap 1 huggingface.co/kamyar-zeinalipour/p-mistral-7b 2 huggingface.co/kamyar-zeinalipour/p-llama2-7b 3 https://huggingface.co/kamyar-zeinalipour/p-llama38b 4 huggingface.co/kamyar-zeinalipour/p-gemma-7b 5 github.com/kamyarzeinalipour/protein-design-llms forward, primarily due to their ability to understand and generate human-like text based on training from extensive datasets. typically, these models are trained using unsupervised learning techniques, where they learn to predict the next word or token in a sequence by examining the tokens that precede it. this method has propelled them to the forefront of various nlp applications, including chatbots @xcite ), text summarization @xcite , and advanced information extraction tasks @xcite . among the intriguing avenues explored with these models is their application in the field of bioinformatics, specifically in protein generation @xcite . indeed, the protein alphabet is composed of twenty common amino acids, each represented by a single character. regarding their primary structure, proteins, which are vital biological molecules, are made up of chains of amino acids, thus forming sequences of letters and drawing a parallel to the structure of natural languages. as in natural languages, protein sequences have directionality and are typically composed of reused modular elements that exhibit slight variations. moreover, common protein motifs and domains, which are the basic building blocks of proteins, are similar to words, phrases, and sentences in human language. this similarity suggests that language models, which excel in handling sequential data, could effectively generate amino acid chains, or proteins. the primary objective of our research lies in advancing the understanding and application of medium-sized language models, particularly those in the 7 billion to 8 billion parameter range, including mistral-7b, llama-2-7b, llama-3-8b, and gemma-7b, for the generation of high-quality protein sequences. our hypothesis, backed by preliminary studies, suggests that these models, even when trained with considerably small datasets, can produce accurate and viable protein sequences effectively. furthermore, we extend our investigation to encompass a comparative analysis utilizing established protein-focused language models such as progen @xcite ; @xcite ), prot-gpt2 @xcite ), and prollama @xcite ). by applying the same experimental conditions across different models, we aim to provide quantitative and qualitative comparisons of their performance and effectiveness. ultimately, this study seeks to validate the capability of medium-sized models in protein design, emphasizing the potential of employing more compact, cost-efficient language models as powerful tools in bioinformatics research. this approach may significantly expedite scientific research and practical applications, spanning from drug design to precision medicine to other interdisciplinary fields. this paper makes the following contributions: the layout of this document is as follows: section 2 reviews previous research. our methods are detailed in section 3. experimental results are discussed in section 4, while conclusions and future perspectives are collected in section 5. ## related works the integration of natural language processing (nlp) techniques, traditionally applied to human languages, into bioinformatics, has transformative potential, particularly in the analysis of biological sequences such as dna, rna, and proteins. these biological data, sharing similarities with linguistic texts in their structured and functional building blocks, are highly amenable to computational methodologies. the impactful success seen in nlp through transformer-based models has led to breakthroughs in specialized models geared toward understanding the complexities of these biological sequences. by utilizing extensive databases such as uniprot (consortium, 2019), ensembl @xcite , these models harness rich data to enhance both predictive and analytical capabilities in bioinformatics. the realm of protein sequences has seen notable advancements through the adoption of both supervised and unsupervised learning models. language models have been increasingly leveraged and employed in the domain of protein design @xcite . supervised learning approaches refine models by training them with labeled data, which is invaluable for accurately predicting protein stability or identifying structural similarities among sequences @xcite . on the other hand, the introduction of transformer technology has been pivotal in popularizing unsupervised learning methods @xcite . these methods involve the strategic corruption of input sequences which are then used to train models to predict and reconstruct the natural sequence. leading models such as esm @xcite , prottrans @xcite , and proteinbert @xcite demonstrate this approach, offering powerful embeddings that prove critical in supporting a wide array of downstream biochemical tasks @xcite . these tasks include, but are not limited to, analyzing protein-protein interactions, predicting molecular functions, and identifying potential sites for drug binding. in addition to these developments, the adoption of autoregressive models -widely recognized for their ability to generate coherent, long-form text in classical nlp settings -has been successfully applied to the domain of protein sequencing. prototypes like progen @xcite , protgpt2 @xcite and prol-lama @xcite capitalize on this capability, employing autoregressive algorithms to effectively predict the future elements of protein sequences from given contexts. this predictive ability is critical for sophisticated applications such as protein design, where the generation of novel and function-ally effective proteins is required. in this study, we employ some pre-trained language models, which we further fine-tune for protein generation tasks, by retraining both the tokenizers and the entire models. we then compare the results with those from other large language models (llms) currently available for protein generation tasks. ## methodology in this section, we delineate the methodologies employed to adapt pre-trained llms for the generation of protein sequences. our approach involved refining the tokenizer based on the byte-pair encoding (bpe) methodology, followed by fine-tuning the entire pre-trained model using a designated dataset of protein sequences. subsequently, this fine-tuned model was utilized to generate new protein sequences. it's important to note that base models such as llms, while powerful, are not inherently capable of designing novel proteins. their success in this domain is achieved through a specialized fine-tuning process, which involves not only adapting the model to a specific task using a smaller, task-specific dataset but also modifying the tokenizer. this is because the tokens that llms were initially trained on are natural language tokens, whereas our domain requires a different set of tokens. therefore, we also need to train the tokenizer to handle this new domain effectively. verification of these sequences was carried out by generating their respective pdb structures using deepmind's alphafold @xcite . we assessed the quality of these structures using various metrics such as plddt, rmsd, tm-score and reu. the performance of the models -namely mistral-7b (jiang et al., 2023), llama-2-7b (touvron et al., 2023), llama-3-8b , and gemma-7b @xcite -was then compared with previous studies that employed language models for protein sequence generation. we have also evaluated the potential fitness of our generated sequences in comparison to natural and random sequences in the context of plddt, rosetta-relax scores, rmsd and tm-scores, thereby providing a comparative analysis. figure 1 illustrates this methodology. subsequently, we will provide a detailed description of all these steps, focusing on the training of the llms and their validation. ## experimental results in this section, we delineate the experiments conducted in this study, presenting an evaluation of the results garnered from the protein sequences we generated. additionally, we discuss the regeneration of proteins utilizing language-based models specifically designed for protein generation tasks, including progen in four distinct sizes, protgpt2, and prollama. initially, we explore the dataset utilized in our experiments, which is notably smaller than those used in other models, followed by a detailed exposition of our training setup. finally, we present a comprehensive analysis of the evaluation results employing various metrics such as plddt, rmsd, tm-score and reu. ## conclusion in this study, we introduced four novel models designed to generate high-quality protein sequences by leveraging pre-trained language models. this research is motivated by the growing demand for efficient and accurate tools that can assist in understanding and engineering protein structures, which are pivotal in numerous biological and medical applications. our approach involved a meticulous design and training phase, followed by rigorous testing and validation processes to assess the performance of each model. to provide a thorough evaluation, we conducted comprehensive experiments comparing our models with a range of existing models that also utilize language models for protein sequence generation. comparative analyses were performed, which were grounded on diverse criteria, including sequence quality, diversity, and fidelity to biological functions. these analyses also incorporated several structural assessment metrics such as plddt (predicted local distance difference), tm-score (to assess structural similarity), rmsd (root mean square deviation), and reu (rosetta energy unit). our findings revealed that some of our proposed models, particularly p-mistral, exhibited superior performance compared to existing models, even surpassing those trained on considerably larger datasets. this remarkable performance underscores the potential of our models to offer significant advancements in the field of protein sequence generation. we are committed to the principles of open science and reproducibility. consequently, we will make all four models publicly available to the research community. this accessibility will empower other researchers to utilize and build upon our work, fostering further advancements in the field of protein sequence generation. moreover, we aim to extend the capabilities of these models by incorporating instruction tuning to generate proteins with specific constraints. this will involve refining the models to adhere to certain criteria, such as ensuring the sequences have particular structural or functional properties. such advancements could be pivotal in various applications, including drug design, synthetic biology, and understanding protein interactions at a deeper level. while our current implementation of llms for protein generation excels in unconditional generation, there is a need to explore and develop methods for generating conditional proteins. this would allow us to guide the generation process toward specific protein characteristics or functions, thereby enhancing the practical utility of our model.