Foundation Model for Biomedical Graphs: Integrating Knowledge Graphs and Protein Structures to Large Language Models (2024)

Abstract: Transformer model has been a de-facto standard in natural language processing. Its adaptations in other fields such as computer vision showed promising results that this architecture is a powerful neural network in representation learning regardless of the data type. This recent success has led to research in multimodal Large Language Model (LLM), which enabled us to new types of tasks and applications with multiple data types. However, multimodal LLM in the biomedical domain is primarily limited to images, text, and/or sequence data. Here I propose to work on multimodal LLM architecture for biomedical graphs such as protein structure and chemical molecules. The research hypothesis is based on the fact that clinicians and researchers in computational biology and clinical research take advantage of various information for their decision-making process. Therefore, an AI model being able to handle multiple data types should boost its ability to use diverse knowledge for improved performances in clinical applications.

## introduction the foundation model revolutionized not only natural language processing (nlp) but also the human-ai interaction after the release of chatgpt service by @xcite . chatgpt enhanced the usability with a chat interface allowing users to instruct large language model (llm) for any tasks even the ones requiring complex domain knowledge such as medical domain text @xcite . the emergence of open-source medical llms has further enhanced access to these technologies in healthcare settings, addressing privacy concerns associated with patient data @xcite @xcite . this success of the foundation model quickly extended to computer vision (cv), expanding the application of chat assistant tools to medical image analytics @xcite @xcite . recently, visual instruction tuning was introduced to open the possibility of a visual assistant in medicine @xcite . additionally, there has been notable progress in extending the model's capabilities to handle biological sequences, including dna sequences and chemical sequences represented by simplified molecular input line entry specification (smiles) notation @xcite @xcite . despite these advancements, multimodal research in biomedicine has focused on integrating text, image, and sequence data. while these modalities have proven invaluable in capturing certain medical nuances, they often overlook the structural intricacies inherent in biomedical graph data, such as knowledge graphs and protein structures. consequently, the full potential of multimodal learning remains largely unexplored in addressing the multifaceted challenges encountered in computational biology and clinical research. ## methods the proposed foundation model for biomedical graphs, depicted in figure 1 , integrates multiple specialized encoders with a backbone llm that is also trained for the biomedical domain to effectively process and understand diverse biomedical data types such as medical images and ontologies. ## conclusion this proposal describes the plan to develop a foundation model architecture uniquely trained to understand the complexities of biomedical graphs. unlike existing models that primarily focus on text, images, or sequences, the proposed model aims to bridge the gap by integrating information from diverse data types such as knowledge graphs, protein structures, and chemical molecules. by leveraging the strengths of large language models in capturing textual information and combining them with specialized encoders for biological sequences and graph structures, the foundation model holds immense potential to revolutionize various aspects of healthcare, including diagnosis, treatment planning, and drug discovery. the model can be used to create an interactive agent that clinicians and researchers can utilize to help them navigate problems in biomedical research, thereby enhancing decision-making processes in clinical practice and computational biology research. for instance, the incorporation of knowledge graphs may allow for a more nuanced exploration of relationships between genes, drugs, and diseases, facilitating target identification for drug discovery as well as drug repurposing, which accelerates the clinical trial progress. moreover, the integration of protein structure and chemical molecule data should enable our model to delve deeper into molecular mechanisms underlying diseases and drug interactions. this deeper understanding opens the possibility of using an assistant tool for more effective protein-drug binding affinity prediction for drug discovery, as well as the identification of potential novel biomarkers for disease diagnosis and prognosis. by leveraging the collective insights from diverse data modalities, the proposed foundation model has the potential to significantly improve performance across a spectrum of biomedical tasks. the development of a multimodal foundation model represents a pivotal step towards unlocking the full potential of artificial intelligence in biomedicine, thereby enhancing our understanding of complex biological systems and ultimately improving healthcare outcomes for patients. moving forward, future work will focus on developing the proposed foundation model to address specific challenges such as training with scarce data. additionally, i will conduct the research with the help of the collective expertise of health infor-matics researchers and clinicians in order to develop the foundation model with a focus on realworld biomedical applications, especially for neurodegenerative diseases.