Can Large Language Models Enhance Predictions of Disease Progression? Investigating Through Disease Network Link Prediction (2024)

Abstract: Large Language Models (LLMs) have made significant strides in various tasks, yet their effectiveness in predicting disease progression remains relatively unexplored. To fill this gap, we use LLMs and employ advanced graph prompting and Retrieval-Augmented Generation (RAG) to predict disease comorbidity within disease networks. Specifically, we introduce a disease Comorbidity prediction model using LLM, named ComLLM, which leverages domain knowledge to enhance the prediction performance. Based on the comprehensive experimental results, ComLLM consistently outperforms conventional models, such as Graph Neural Networks, achieving average area under the curve (AUC) improvements of 10.70% and 6.07% over the best baseline models in two distinct disease networks. ComLLM is evaluated across multiple settings for disease progression prediction, employing various prompting strategies, including zero-shot, few-shot, Chain-of-Thought, graph prompting and RAG. Our results show that graph prompting and RAG enhance LLM performance in disease progression prediction tasks. ComLLM exhibits superior predictive capabilities and serves as a proof-of-concept for LLM-based systems in disease progression prediction, highlighting its potential for broad applications in healthcare.

## introduction the digital transformation of healthcare through artificial intelligence (ai) has reshaped health management @xcite . electronic health records now provide a rich source of data for predictive analytics, improving patient care by forecasting outcomes such as mortality rates @xcite , length of stay @xcite , and disease progression @xcite . nevertheless, traditional methods relying on static data and uniform standards are insufficient for addressing individual patient needs @xcite . this presents a significant challenge in utilising this extensive data for proactive health management. this challenge extends into the crucial areas of disease progression and comorbidity prediction in healthcare research @xcite . comorbidity, or the co-occurrence of multiple health conditions, not only complicates clinical management but also leads to worse health outcomes and higher healthcare costs @xcite . conditions like arthritis and cardiovascular disease often coexist, emphasising the need for predictive models that focus on the relationships between comorbidities rather than solely on lab results @xcite . to enhance predictive accuracy for disease comorbidity, researchers have developed various methods. @xcite created a comorbidity network and applied link prediction techniques to forecast chronic diseases based on patients' current health statuses. however, heuristic link prediction techniques can lack precision, scalability, and robustness, often failing to consider the features of the nodes and edges in the network @xcite . @xcite analysed hypertension comorbidities using network analytics but found that their method was not robust or generalisable, relying heavily on specific datasets. recently, @xcite introduced a framework utilising graph learning to explore chronic disease comorbidity and progression patterns. nevertheless, this approach faces challenges, including dataset acquisition costs and generalisability issues. large language models (llms), defined as advanced ai systems capable of processing and generating human-like text based on vast amounts of training data @xcite , have demonstrated exceptional capabilities across various nlp tasks in medicine @xcite ; xie et al., 2023). despite these successes, applying llms in predicting disease progression and transforming medical data into actionable narratives is largely underexplored. to address the gap in disease progression research, we introduce a framework for disease comorbidity prediction using llm, named comllm, that utilises llms to extract and analyse disease networks. this method capitalises on the extensive clinical knowledge, employing llms to predict disease progression within disease networks. figure 1 illustrates a framework designed to predict potential disease progression within a network using an llm. the framework starts by mapping diseases as nodes within a network, linked by known relationships. for each disease, such as alzheimer's, comprehensive details including descriptions and 1-hop neighbours are utilised in a link prediction task to identify possible new relationships between diseases. this task incorporates a retrieval-augmented generation (rag) strategy, which enriches the model's predictions by integrating relevant domain knowledge. the llm processes prompts that incorporate graph information to describe potential relationships between diseases and assesses the likelihood of a link, ultimately providing a binary response to indicate the presence or absence of a connection. this framework aims to significantly improve the accuracy of disease relationship predictions by combining disease network information with advanced nlp techniques. specifically, the system uses two disease networks, the human disease network @xcite and the human symptoms-disease network @xcite , with different graph sizes to compare the llm's performance using vari-ous prompts and strategies. our system incorporates automated disease feature generation from gpt-4. additionally, the framework utilises the langchain (langchain, 2024) framework to retrieve domain knowledge via rag. finally, the system is tested with different prompts to improve prediction accuracy. through comprehensive experiments across various settings (zero-shot, fewshot, chain-of-thought (cot), graph prompt and rag), we observed that comllm surpasses numerous baseline models, and when integrated with rag, it exceeds the performance of graph neural network (gnn)-based models @xcite in link prediction tasks. the experimental outcomes indicate that comllm, with the rag, records average auc enhancements of 10.70% and 6.07% over the best-performing model in baselines in the human disease network and the disease-symptoms network, respectively. additionally, our method achieves these results with considerably fewer disease records and information, offering strong support for integrating open-world knowledge into healthcare prediction. our main contributions include: large language models in disease prediction: llms have demonstrated impressive capabilities across several natural language processing tasks in the healthcare domain. however, their potential to predict disease progression and convert medical data into actionable narratives remains untapped. @xcite proposed a framework called coad, which highlights the ability of llms to analyse health reports and assess medical conditions. another significant advancement is the clinical prediction with large language models (cpllm) method, introduced by shoham and rappoport (2023), which leverages historical diagnosis data and outperforms traditional logistic regression and advanced models like med-bert @xcite in predicting future disease diagnoses. furthermore, @xcite developed the health-llm framework, combining feature extraction with medical knowledge scoring to enhance disease prediction and personalise healthcare. @xcite introduced the graph-care framework, which merges external knowledge graphs with electronic health records via a novel bi-attention augmented gnn, significantly improving key healthcare outcomes such as mortality, readmission, length of stay, and drug recommendation. although these methods represent significant progress in disease diagnosis prediction, they have yet to completely unravel the complex interrelations among diseases, which is crucial for further enhancing their efficacy in predicting dis-ease progression. llms in disease prediction often face limitations such as generating outputs that may be factually incorrect or irrelevant, particularly when dealing with complex data sets in healthcare @xcite . this is where rag becomes crucial. rag addresses these limitations by integrating external, verified information, thus enhancing the model's ability to produce accurate and relevant responses. for instance, in the health-llm framework, rag uses external medical databases to refine its predictions and reduce the likelihood of erroneous outputs, ensuring that the advice and diagnostics provided are based on the most current and comprehensive data available @xcite . this integration is vital in healthcare, where the accuracy of information can significantly impact patient outcomes. link prediction on disease networks: initial studies on link prediction within disease networks relied on heuristic techniques that calculated node similarity based on the graph's structure to predict potential links from these similarity scores. however, these methods primarily focused on the structural properties of the graph, neglecting individual node features and the broader domain knowledge associated with each node @xcite . advanced ml algorithms, including gnns and matrix factorisation, have since been utilised to glean deeper insights from complex disease networks @xcite . while gnns effectively capture hierarchical and non-linear structures within the network, they often face challenges with complex medical data due to insufficient integration of external domain knowledge. building on previous research, our work uniquely applies llms to link prediction within disease networks. this approach surpasses traditional methodologies by dynamically integrating extensive medical knowledge, enabling a more precise understanding of disease progression. ## experimental settings baselines: we assess the performance of our proposed model by comparing it with leading link prediction methods. initially, we employ heuristic approaches, such as common neighbour (cn) and adamic-adar index (aa) @xcite , which have been widely used in prior research @xcite . subsequently, we explore more advanced methods, such as matrix factorisation (mf), which integrates latent and explicit features within the graph. in this method, each node is assigned various embeddings, which are trained end-to-end using a multilayer perceptron (mlp) predictor @xcite . we use an embedding size of 64 and train for 100 epochs for this method. following this, we implement node2vec, a method that maps nodes into an embedding space @xcite . in this study, these embeddings are utilised in a downstream link prediction task using a three-layer mlp. the dimensions are set to 64, the walk length to 30, and the number of walks to 200. additionally, we benchmark our model against the latest gnn-based models. graph convolutional network (gcn), a type of gnn, utilises convolutional neural networks on graphs (kipf and welling, 2016), while graphsage is another gnn variant designed for inductive representation learning on large graphs @xcite . moreover, contrastive multi-view representation learning on graphs (mvgrl) employs a self-supervised approach to learn representations by contrasting structural graph views @xcite , and learning from subgraphs, embeddings, and attributes for link prediction (seal) uses gnns in a deterministic manner, leveraging node features and hand-crafted labels @xcite . furthermore, we include recent gnn-based disease progression models, cpllm (shoham and rappoport, 2023) and health-llm @xcite , as well as the llm-based disease prediction model, graphprompter @xcite , for comparison. for these baselines, we used default settings for the hyperparameters in dgl @xcite . evaluation setting: the focus of this study is on link prediction, defined as a binary classification challenge. in this context, the inputs to the model are pairs of nodes. to construct these inputs, we split the positive edges of our network into training and testing sets. during the training phase, we generate an equivalent number of negative edges through random sampling. specifically, for the dis-ease network, we allocate 10% of the existing edges to the train set as positive samples and another 10% to the test set while also creating an equal number of non-existent edges as negative samples. in the case of the broader disease symptoms network, we use 10% of the existing edges for training and 1% for testing, corresponding to 11,311 positive edges, and similarly generate an equal number of non-existent edges for negative samples. for the llms, we utilise gpt-3.5 and gpt-4-turbo from openai (openai, 2024b), llama 2, llama 3 and llama 3.1 from meta ai (meta, 2024). gpt-3.5 acts as a transitional model between gpt-3 and gpt-4, providing enhanced reasoning and reduced computational needs. gpt-4 expands on this with more advanced algorithms and a larger training dataset, significantly improving accuracy and knowledge breadth. llama 2, llama 3 and llama 3.1, developed by meta ai, offer a scalable and efficient architecture, allowing customisation to match specific computational limits and application requirements. these models are instrumental in advancing ai applications across various fields, reflecting rapid innovations and the increasing sophistication of language models. the specific models we use for gpt-3, gpt-4, llama 2, llama 3, and llama 3.1 are gpt-3.5-turbo, gpt-4-turbo, llama-2-7b-hf, meta-llama-3-8b and meta-llama-3.1-405b-instruct, respectively. we set the temperature to zero and the maximum token to 64. in assessing the performance of models for link prediction tasks, we use the area under the roc curve (auc), average precision (ap), and f1 score metrics. the auc represents the area under the receiver operating characteristic (roc) curve, which depicts the true positive rate versus the false positive rate at different classification thresholds @xcite . the testing case sets the threshold at a balanced 50% true positive rate and 50% false positive rate, given an equal number of positive and negative data points. a higher auc value suggests that the model is more effective at distinguishing between positive and negative examples. the ap metric summarises the precision-recall curve as the weighted average of precision at each threshold, using the increment in recall from the previous threshold as the weight @xcite . this metric is particularly useful for evaluating model performance in terms of precision (accuracy of positive predictions) and recall (capacity to identify positive instances), especially in cases of class imbalance. the macro f1 score, which ranges from 0 to 1, reflects the balance between precision and recall, with higher scores indicating better performance @xcite . we ran all the baselines and llms, testing one variation for each prompt example, and averaged the performance results over five runs; the numbers in brackets represent the standard deviation. the baseline graph machine learning methods are executed using the dgl @xcite and networkx @xcite libraries in python. the experiments are conducted on a server equipped with an a40 gpu with 48 gb of memory. ## conclusion in this study, we introduce comllm, a framework that merges graph prompts and rag with llms for improved disease progression predictions in disease networks. empirical evidence demonstrates its superiority over baseline models, such as gnnbased models, across two datasets with different graph sizes. our framework can predict potential disease comorbidities effectively. beyond its practical improvements, comllm also acts as a proofof-concept for llm-based systems in healthcare, underscoring the significant potential for further developing ai applications in the health sector.