Evaluating Large Language Models for Predicting Protein Behavior under Radiation Exposure and Disease Conditions (2024)

Abstract: The primary concern with exposure to ionizing radiation is the risk of developing diseases. While high doses of radiation can cause immediate damage leading to cancer, the effects of low-dose radiation (LDR) are less clear and more controversial. To further investigate this, it necessitates focusing on the underlying biological structures affected by radiation. Recent work has shown that Large Language Models (LLMs) can effectively predict protein structures and other biological properties. The aim of this research is to utilize open-source LLMs, such as Mistral, Llama 2, and Llama 3, to predict both radiation-induced alterations in proteins and the dynamics of protein-protein interactions (PPIs) within the presence of specific diseases. We show that fine-tuning these models yields state-of-the-art performance for predicting protein interactions in the context of neurodegenerative diseases, metabolic disorders, and cancer. Our findings contribute to the ongoing efforts to understand the complex relationships between radiation exposure and disease mechanisms, illustrating the nuanced capabilities and limitations of current computational models.

## introduction the exploration of the biological consequences of ionizing radiation on human health has long been a focal point of medical and environmental research. high doses of radiation are linked to immediate cellular damage and an increased risk of cancer @xcite . however, the implications of low-dose radiation (ldr) exposure remain a topic of significant debate. emerging evidence suggests potential associations with various non-cancerous diseases, including neurodegenerative and cardiovascular diseases @xcite . additionally, others show that cancer is a result of low-dose radiation exposure @xcite . understanding these effects at the molecular level, particularly in relation to protein structure and function, is crucial for developing protective measures. similarly, protein-protein interactions (ppis) are vital for various cellular processes and play a critical role in understanding disease mechanisms. furthermore, there exists extensive ppi data, compiled into comprehensive public databases like biogrid @xcite , string @xcite , hippie @xcite , and kegg @xcite . considerable research has been dedicated to understanding general protein interactions; however, there is a lack of studies examining protein interaction networks in the context of specific diseases. the overarching goal of this research is to determine the efficacy of llms in accurately predicting complex biological processes related to protein function under various conditions. we employ three state-of-the-art llms, to analyze data from six diverse datasets. these datasets represent two distinct categories. the first focuses on the effects of ldr on proteins, and the second highlights the ppi network present within specific diseases. we formalize this data into two binary classification tasks, which are illustrated in figure 1 . this approach not only demonstrates the versatility of llms in biological research but also paves the way for novel insights into the molecular dynamics influenced by radiation exposure and disease processes. our contributions in this paper include: ## llms and datasets in this study, we employ three open-source llms, mistral (7b), llama 2 (7b), and llama 3 (8b), to investigate two primary areas of biological research: the effects of low-dose radiation (ldr) on proteins and the dynamics of ppis in the context of specific diseases. these models were chosen because of their state-of-the-art performance in many natural language processing (nlp) tasks. additionally, their open-source nature allows for broad accessibility and modification by researchers across disciplines, which promotes transparency and collaborative advancements in both nlp and other scientific domains. to facilitate a comprehensive analysis, our methodology encompasses six core datasets, which are further subdivided into 13 distinct subsets based on specific experimental parameters and objectives. the first 3 datasets primarily explore the effects of ldr on protein deregulation. these 3 sets are further divided into 10 subsets, emphasizing different experimental conditions. the last 3 datasets focus on ppis in the presence of specific diseases, namely neurodegenerative, metabolic, and cancer. the subsets of the ldr data are much smaller than the ppi datasets, which is why these were combined into dataset 3c. dataset 3c's larger size is shown in comparison with the other datasets in figure 2 . the details about each dataset is outlined in appendix a. ## experiments the methodology for analyzing each dataset began with data pre-processing, executed through a python script tailored to appropriately structure the raw data. subsequently, this processed data was used to create prompts that fit the prompting strategies outlined in appendix a. these prompts were then saved in a json file, and were subsequently used as input to the llms. for deploying the models, a separate python script using the hugging face transformers library loaded the models onto 4×nvidia a100 80gb gpus. these pre-trained models were then presented with the json file prompts and the performance of each model was recorded. ## results every phase of the model training process was documented and analyzed. the evaluation metrics used include accuracy, matthews correlation coefficient (mcc), specificity, macro precision, and macro f1 score. tables 1-13 indicate the performance of both the pre-trained models, and their fine-tuned counterparts on each of the 13 datasets. we evaluated the pre-trained models using the same procedure as the fine-tuned models, the only difference is that the ## evaluation of model predictions in this section, we analyze the predictions made by the llms and highlight some of the proteins that were correctly and incorrectly identified. we focus on interpreting the results obtained from our experiments in tables 1-13, examining the predictions made and identifying patterns in each model's output to understand their current limitations. ## dataset cross-reference analysis independent of the llm experiments, we conduct a dataset cross-reference analysis to identify the common proteins between the ldr and ppi datasets, highlighting those that may be involved in both processes. through this extensive analysis, we gained a deeper understanding of the data utilized for training these llms and enhanced our understanding of the protein dynamics involved in both radiation response and disease mechanisms. we identified overlaps between the ppi datasets 4, 5, and 6, and the combined ldr dataset 3c. the positive interaction pairs were identified for each of datasets 4 (11,762 proteins), 5 (10,262 proteins), and 6 (1,866 proteins). subsequently, the significantly affected proteins in the combined dataset 3c were identified (1,111 proteins). our findings show that the highest percentage of overlap with the ldr data was with dataset 4, the neurodegenerative ppi dataset. ## conclusion this study presents an exploration of the capabilities of llms in predicting the molecular dynamics of proteins under various conditions. by employing three state-of-the-art llms across multiple datasets, our research offers valuable insights into the potential utility and limitations of computational models for these tasks. the fine-tuning process using lora proved to be a pivotal factor in enhancing model performance, demonstrating notable improvements in accuracy and predictive capabilities. improving the accuracy of these models is key, because a major limitation of llms is their tendency to hallucinate, or give false information. utilizing parameter efficient finetuning strategies helps to alleviate this problem while also maintaining an efficient computational complexity. through the use of peft, the llama 3 model outperforms the current best models for the ppi prediction tasks, indicating its potential for future advancements in biomolecular research. our analysis of protein identification by these models revealed intriguing patterns. correctly identified proteins often belonged to well-characterized functional categories and were represented by standard naming conventions, suggesting that the pretraining on extensive biomedical literature may have equipped the models with a robust foundation of biological knowledge. conversely, proteins that were incorrectly identified typically lacked these characteristics, possibly indicating areas where llms could benefit from further training or more focused dataset enrichment. the cross-referencing of proteins affected by ldr with those involved in ppis of neurodegenerative, metabolic, and cancer-related processes brought forth specific proteins that could be further explored in future studies. notably, the neurodegenerative ppi dataset showed the highest overlap, where 115 unique proteins were identified in both datasets. these results highlight exactly which proteins in the ppi networks are significantly deregulated after ldr exposure, which could help to advance our understanding of how ldr affects disease mechanisms. in conclusion, the integration of llms into biological research, particularly using fine-tuning techniques like lora, holds promising potential for advancing our understanding of the molecular mechanisms underpinning disease and radiation exposure. the versatility and scalability of these models make them instrumental tools in the ongoing quest to decode complex biological data. their capacity to learn patterns and generate insights from extensive datasets holds immense promise for future research endeavors. future work should focus on expanding the datasets, specifically the ldr data, and refining model architectures to further enhance the precision and applicability of llms in scientific discovery.