Can LLM s Replace Clinical Doctors? Exploring Bias in Disease Diagnosis by Large Language Models (2024)

Abstract: The bias of disease prediction in Large Language Models (LLMs) is a critical yet underexplored issue, with potential implications for healthcare outcomes and equity. As LLMs increasingly find applications in healthcare, understanding and addressing their biases becomes paramount. This study focuses on this crucial topic, investigating the bias of disease prediction in models such as GPT-4, ChatGPT, and Qwen1.5-72b across gender, age range, and disease judgment behaviors. Utilizing a comprehensive real-clinical health record dataset of over 330,000 entries, we uncover that all three models exhibit distinct biases, indicating a pervasive issue of unfairness. To measure this, we introduce a novel metric–the diagnosis bias score, which reflects the ratio of prediction numbers to label numbers. Our in-depth analysis, based on this score, sheds light on the inherent biases in these models. In response to these findings, we propose a simple yet effective prompt-based solution to alleviate the observed bias in disease prediction with LLMs. This research underscores the importance of fairness in AI, particularly in healthcare applications, and offers a practical approach to enhance the equity of disease prediction models.

## introduction in recent years, the rapid advancement of artificial intelligence technologies, particularly large language models (llms), has significantly impacted disease diagnosis and clinical decision support systems @xcite @xcite . automatic disease prediction, which uses a patient's medical notes such as electronic medical records (emr), aims to predict the most likely diseases, aiding doctors in making accurate figure 1 : illustration of the count of predicted and actual cases of "meniscus injury" in both male and female records, as determined by gpt-4, chatgpt, and qwen. "m-pd" and "m-ad" stand for "male predicted disease" and "male actual disease" respectively, while "f-pd" and "f-ad" represent "female predicted disease" and "female actual disease", respectively. clinical decisions. timely and precise disease prediction can facilitate early intervention, optimize disease management and improve the efficiency of healthcare resource allocation. however, llms, trained on extensive text corpora, have been found to exhibit notable levels of social biases @xcite @xcite @xcite . these unchecked biases could potentially perpetuate and amplify diagnostic errors as llms become increasingly competent and start to serve as integral components in healthcare decision-making systems. we analyzed over 330,000 authentic health records and discovered apparent disparities between the distributions of predicted disease samples and the actual samples for both males and females. for instance, figure 1 illustrates the count of predicted and actual instances of "meniscus injury" in both male and female records, as diagnosed by @xcite qwen. the contrast between dark green and light green, as well as between dark orange and light orange, signifies the discrepancies between the predicted and actual counts of "meniscus injury" records. this clearly demonstrates a bias in disease diagnosis with large language models (llms). specifically, there are 280 male and 600 female patients with meniscus injuries. however, gpt-4 predicted 440 (out of the whole dataset) men with meniscus injuries, of which only 80 actually had meniscus injuries, while the rest had joint diseases. therefore, the model's accuracy for predicting meniscus injuries in males is approximately 18.2%. for females, gpt-4 predicted 460 cases, with only 140 actually having meniscus injuries and the rest having joint diseases, resulting in an accuracy of about 30%. in reality, the model is more inclined to diagnose men with meniscus injuries (predicting 440 cases compared to the actual 280), while it underdiagnoses women (predicting 460 cases compared to the actual 600). if we only look at accuracy, females (30%) appear to have a higher accuracy than males (18%), which does not reflect the model's tendency to overdiagnose men with this condition. in addition to the explicit bias observed in the generated reasoning sections, we also identify implicit gender biases in disease prediction using language model libraries (llms). specifically, our health records indicate an equal number of male and female myocardial infarction (mi) cases, each totaling 600. however, only 340 mi cases were predicted for females, compared to 680 for males, suggesting a stricter diagnostic threshold for women. this observation aligns with @xcite , which found that the focus on male symptoms in heart attack research often leads to the overlooking of differing symptoms in women, resulting in higher mortality rates due to medical negligence. moreover, diagnostic bias extends beyond just gender and age. for instance, llms tend to "copy" the disease name mentioned in records, potentially overlooking more critical diagnoses. for example, if a record mentions high blood pressure, the model will likely diagnose hypertension. additionally, llms often provide more severe diagnoses. for instance, if a doctor diagnoses gastritis, llms typically diagnose it as stomach cancer. given that bias manifests across multiple dimensions, we find it necessary to introduce a metric to measure fairness across these various dimensions. therefore, in this paper, we introduce a new metric-the diagnosis bias score-to quantify the level of disease prediction bias in llms. the fundamental insight of the diagnosis bias score is that the higher the ratio of the model's predicted sample size to the actual disease sample size for a particu-lar disease, the more the model tends to diagnose that disease. we measure the model's bias based on the differences in this tendency across various dimensions. for example, in the gender dimension, we examine the difference between the model's predicted quantity of a disease in males and females and the actual sample size of the disease in males and females to gauge the model's gender bias for that disease. to mitigate bias in disease prediction by large language models, we introduce an integrated debiasing diagnosis method (idd). this method involves concealing gender and age information to prevent biased diagnoses, and it alerts the model to the potential for overdiagnosis and the tendency to repeat disease names in records. specifically, we obscure unique information related to gender and age, one at a time, and input the remaining data into the llms for the initial round of disease prediction. subsequently, we feed the dimensionagnostic predicted diseases back into the llms, triggering a second round of disease prediction. simultaneously, the model is encouraged to evaluate whether the diagnostic result is supported by sufficient evidence, thereby avoiding overdiagnosis or insufficient reasoning by merely replicating disease names from records. ultimately, the model reassesses the four dimension-unbiased disease candidates to reach a final diagnostic conclusion. this approach guides the llms to make decisions both with and without potential biases from different dimensions, thereby promoting more accurate and unbiased disease diagnoses. in summary, we make the following contributions in this paper: for hard-to-diagnose and rare diseases, medical knowledge has been used to enhance contrastive learning for few-shot disease diagnosis @xcite . ## integrated debiasing diagnosis our proposed debiasing diagnostic method can guide models towards an unbiased diagnosis by focusing on four key dimensions: gender, age, disease-severity, and record repetition. as illustrated by figure 11 , initially, the model is tasked with predicting diseases from a medical record where gender and age information are deliberately obscured. subsequently, the model is required to make a gender-unbiased and age-unbiased diagnosis based on the preceding gender-agnostic and age-agnostic diagnosis results. regarding disease severity, the model predicts the disease from the record and evaluates if the predicted disease is overdiagnosed. for the record repetition dimension, the model first identifies all disease names within the record and then makes a disease prediction that avoids merely replicating the mentioned disease names. ultimately, the model reassesses the four disease candidates, one from each dimension, to reach a final, unbiased diagnostic conclusion. detailed instruction templates for idd can be found in tables 9 10 11 12 . ## conclusion this paper addresses the bias inherent in disease prediction using large language models (llms). to quantitatively assess this bias, we introduce a new metric-the bias score-which measures the bias across different dimensions of disease prediction. utilizing this metric, we conduct an indepth analysis of bias from four critical dimensions: gender, age, disease severity, and record repetition, across over 330,000 health records. furthermore, to mitigate the bias in disease prediction with llms, we propose an innovative, integrated debiasing diagnostic method. furthermore, the outcomes of a series of meticulous experiments offer compelling empirical evidence in support of our methods.