Geneverse: A Collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Research (2024)

Abstract: The applications of large language models (LLMs) are promising for biomedical and healthcare research. Despite the availability of open-source LLMs trained using a wide range of biomedical data, current research on the applications of LLMs to genomics and proteomics is still limited. To fill this gap, we propose a collection of finetuned LLMs and multimodal LLMs (MLLMs), known as Geneverse, for three novel tasks in genomic and proteomic research. The models in Geneverse are trained and evaluated based on domain-specific datasets, and we use advanced parameter-efficient finetuning techniques to achieve the model adaptation for tasks including the generation of descriptions for gene functions, protein function inference from its structure, and marker gene selection from spatial transcriptomic data. We demonstrate that adapted LLMs and MLLMs perform well for these tasks and may outperform closed-source large-scale models based on our evaluations focusing on both truthfulness and structural correctness. All of the training strategies and base models we used are freely accessible. Our codes can be found at https://github.com/HelloWorldLTY/Geneverse .

## introduction foundation models (fms) @xcite have attracted great attention recently because of their superb functionality for handling multiple downstream tasks by adaption, especially in the landscape of natural language processing (nlp) @xcite . in this area, we focus on one specific type of fms, known as generative large language models (llms) @xcite . llms have demonstrated their strong ability to perform various tasks and domains in nlp @xcite ), * corresponding author. and they have revolutionized the approaches of productivity improvement with machine intelligence @xcite as well. powerful llms including chatgpt @xcite and bard @xcite exhibit the ability to understand and reason as a human, and they are capable of complicated problem-solving tasks. however, since these well-trained models are both closed-source, the emergence of open-source llm development is also important. powerful open-source llms including llama2 @xcite and @xcite have already become the base models of many llms for scientific research @xcite . beyond the success of llms, we are also interested in extending the input of llms with datasets from different modalities, including images @xcite and scientific graphs @xcite . such extensions of llms are meaningful because different modalities offer different aspects of a good representation @xcite . therefore, multimodal llms (mllms) are emerging as a new research hotspot @xcite . models like gemini (closed-source) @xcite , gpt-4 (closed-source) @xcite and llava (open-source) @xcite allow the combination of images and texts as input, thus making mllms understand non-text data and generate responses accordingly @xcite . the performance of these models demonstrates that treating images as a special language can extend the capabilities of llms to solve more tasks including image-based conversation, image description, and complex reasoning @xcite . although we have seen successful examples related to the development of llms and mllms, the application of llms towards healthcare and biomedical research presents both opportunities and challenges @xcite . first, different from other areas, biomedical research highlights strict requirements for the factualness or truthfulness of model outputs @xcite , as well as the need to validate outputs by clinical or biological experiments @xcite . when llms are used to assist physicians in the diagnosis of patients, an incorrect diagnosis of a patient's condition can dramatically affect the process and even cause the patient to miss the optimal time for treatment. second, the output of llms should comply with established scientific knowledge @xcite . third, the performance of finetuned open-source llms has not outperformed large or proprietary models @xcite for such research. regarding the opportunities, most of the llms for biomedical research focus on precision healthcare @xcite and are finetuned with information from the corpus of biomedical articles @xcite , while the llms or mllms for genomic or proteomic research are not welldiscussed and remain to be explored. therefore, we aim to contribute such fms to accelerate research related to the central dogma @xcite at the molecular level @xcite . to fill the gap in this area, we propose a collection of llms and mllms, named as geneverse foot_0 , for solving genomic and proteomic problems. our collection mainly focuses on the functions of genes and proteins, as well as their interactions. overall, our contributions can be summarized as follows: ## discussion foundation models, especially llms, are powerful tools for scientific research. in this manuscript, we introduced geneverse, a collection of llms and mllms tailored for genomic and proteomic tasks. we compared different strategies and base models for generating accurate function descriptions of genes based on the adapted llms. moreover, we also explored the applications of mllms for a protein-level task and a gene-level task, as an example of leveraging the multimodal information. the models in geneverse demonstrated better performance in the selected tasks than closed-source models, thus advocating the contributions of opensource base models for scientific research. we also discussed the factors affecting model adaptation, which advanced this field by offering suggestions. our future work aims to increase the generation quality of outputs from geneverse with more advanced techniques. we will also investigate why rag-based llms fail and improve them. moreover, we will collaborate with other institutes to access more high-quality datasets for model adaptation. the next-generation multimodal ai assistant for genomics and proteomics will come soon. ## limitations first, finetuning an fm or llm requires a large amount of computing resources. based on our testing results, we need at least one h100 or equivalent gpu cores to perform finetuning with lora. secondly, to utilize methods based on closed-source large models, including gpt series and genegpt, we need to use the openai api to call the method, thus we need additional resources for deploying such a method. finally, a good domain-specific fm should align with the version of state-of-the-art base models. therefore, we should keep our model updated by aligning the best version of geneverse with the best open-source base model. moreover, due to the limitation of computing resources, we only considered 7b-level and 13b-level models in the current version. in the future, we need to explore llms or mllms with larger scales. finally, our evaluations for the domain-specific fms only focus on three tasks and more tasks should be included in the future. ## ethics statement the users are solely responsible for the content they generate with models in geneverse, and there are no mechanisms in place for addressing harmful, unfaithful, biased, and toxic content disclosure. any modifications of the models should be released under different version numbers to keep track of the original models related to this manuscript. the target of current geneverse only serves for academic research. the users cannot use it for other purposes. finally, we are not responsible for any effects of the use of the model. ## codes and reproductivities the codes and datasets we used in this manuscript can be found in https://github. com/helloworldlty/geneverse . we used the mit licenses. all of the model weights can be accessed via requests. we used two nvidia h100 gpu cores for model training, while the maximum system memory for training is 100 gb. gpt 4 also has the ability to handle image data as input. genegpt is based on searching documents to enrich the information in the prompts. we use the openai api to access the base model of this method. for mllms, we have the following models: q: what is the name of this protein? please summarize your answer in one sentence. a: it is af-o43280-f1. instruction example lmm q:is gene apoc1 a marker gene of cell type macrophage? please summarize your answer in one sentence. a: yes. q: please summarize the major function of gene: gli1. use academic language in one paragraph and include pathway information. a: the official symbol for gli1 is provided by hgnc, and its official full name is gli family zinc finger 1. the primary source for this information is hgnc:hgnc:10997, and it is also related to ensembl:ensg00000104844 and mim:600619; alliancegenome:hgnc:10997. this gene is a protein coding gene and its refseq status is reviewed. it is found in the organism homo sapiens and belongs to the lineage eukaryota; metazoa; chordata; craniata; vertebrata; euteleostomi; mammalia; eutheria; euarchontoglires; primates; haplorrhini; catarrhini; hominidae; homo. it is also known as gli; gli-1; gli-2; gli-3; gli-4; gli-5; gli-6; gli-7; gli-8; gli-9; gli-10; gli-11; gli-12; gli-13; gli-14; gli-15; gli-16; gli-17; gli-18; gli-19; gli-20; gli-21; gli-22; gli-23; gli-24; gli-25; gli-26; gli-27; gli-28; gli-29; gli-30; gli-31; gli-32; gli-33; gli-34; g a: the official symbol for gli1 is provided by hgnc, and its full name is gli family zinc finger 1. the primary source for this information is hgnc, and it is also related to ensembl and mim. gli1 is a protein coding gene in the organism homo sapiens and belongs to the zinc finger family of transcription factors. it is involved in the hedgehog signaling pathway, which plays a crucial role in embryonic development and adult tissue homeostasis. gli1 is a transcriptional activator of hedgehog target genes and is regulated by post-translational modifications. it is expressed ubiquitously in various tissues, including the esophagus, skin, and 25 other tissues. it also has orthologs in mice.