Large Language Models Encode the Practice of Medicine (2024)

Abstract: Healthcare tasks such as predicting clinical outcomes across medical and surgical populations, disease prediction, predicting patient health journeys, are typically approached with supervised learning on task-specific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of billions of administrative claims, which essentially encapsulates the practice of medicine, offering a unique perspective on patient care and treatment patterns. Our model, MediClaimGPT, a 125M parameter Transformer demonstrates strong zero-shot predictive capabilities, accurately forecasting patient health events across four evaluation datasets, with its capabilities further demonstrated in various downstream tasks. A significant application of MediClaimGPT is in generating high-quality, clinically plausible synthetic claims data, enhancing healthcare data utility while preserving patient privacy. This research underscores the potential of language models in handling complex datasets and their strategic application in healthcare and related fields.

## introduction administrative claims data, a crucial component of the healthcare sector, adeptly captures the intricacies of the practice of medicine. it provides extensive coverage @xcite , capturing detailed patient histories through insurance reimbursement records. these data, rich in diagnostic and procedural information encoded in medical codes like icd-10-cm @xcite and cpt @xcite , are pivotal in understanding healthcare delivery and patient care patterns (see appendix a for more details). however, their complexity challenges traditional data processing, necessitating innovative ai approaches @xcite . the emergence of large language models (llms) signifies a transformative phase in data analytics, particularly within the healthcare sector, where their ability to process vast, unstructured datasets has groundbreaking potential @xcite . while language models like biobert @xcite , scibert @xcite , pub-medbert @xcite , and clinicalbert @xcite have excelled in biomedical nlp tasks, and conversational models such as med-palm @xcite , med-palm 2 @xcite , chatdoctor @xcite , and baize-health @xcite have shown impressive results in medical questionnaires, they exhibit limitations in fully grasping the practice of medicine and predicting clinical outcomes. these models, despite their advancements, often lack the depth of understanding needed to accurately predict patient-specific clinical outcomes, a key aspect in the realm of medical practice and decision-making support. codes: the first row illustrates vaccine sequence prediction (covid-19 vaccine dosages) and the second demonstrates surgical likelihood assessment for spinal conditions. these examples highlight mediclaimgpt's capacity in zero-shot settings to generate clinically relevant predictions. our model, mediclaimgpt, aims to bridge this gap, it is uniquely trained on a vast dataset of 70m patients and 3b claims, focusing on the comprehensive healthcare journey of each patient. by structuring this dataset to represent each patient as a sequence of medical claims, encoded as medical codes, mediclaimgpt is tailored for medical practice intricacies. its performance in zero-shot scenarios and various downstream tasks highlights its broad utility in healthcare data analytics. a key breakthrough of mediclaimgpt is its application in generating synthetic claims data that closely mirrors real data's statistical properties while ensuring anonymity, addressing privacy concerns in line with hipaa guidelines @xcite . this innovation not only aids in balancing data disparities but also enhances the scope of healthcare research within privacy compliance frameworks @xcite . while each medical code has an associated english description, we opted to use only the codes themselves. this decision was driven by the observation that converting codes in the claims to descriptions often disrupts textual coherence, leading to disjointed sentences and a lack of semantic flow. moreover, using descriptions significantly increases the context length. for instance, converting a year of a patient's health history into descriptions resulted in an average sequence length of 32k tokens using the tiktoken library. considering that clinical event prediction typically requires more than two years of data, the sequence length becomes impractically long. additionally, in zeroshot settings where the model predicts health outcomes from a patient's history (see in this paper, we present how llms like medi-claimgpt can effectively manage and process complex healthcare data, setting a new benchmark in healthcare analytics. our contributions are as follows: the rest of the paper is organized as follows. we review related work in section 2. our approach for training mediclaimgpt is described in section 3. the experiments and evaluations are detailed in section 4. finally, we conclude the paper in section 5, reflecting on the significant impact and potential of our work in transforming healthcare data analytics. ## related work the application of machine learning to administrative claims data have been explored in various studies. @xcite demonstrated the potential of claims data in predicting clinical outcomes across medical and surgical populations, while @xcite @xcite focused on identifying high-cost patients. @xcite leveraged this data for disease prediction. @xcite calculated in-hospital mortality using claims data, highlighting the versatility of machine learning in handling various facets of healthcare. certain studies in @xcite @xcite , utilized diagnosis codes from ehrs and employed advanced neural network methods for clinical event prediction. representation learning methods have also been explored @xcite , with techniques ranging from bert to stacked denoising encoders to model ehr data. @xcite proposed direct prediction of diagnosis and procedure codes from ehr. however, these ehr-based approaches face limitations due to data inconsistency and sparse availability @xcite . while @xcite attempted to harness external knowledge bases to augment insufficient ehr data for disease prediction, it still suffers from low coverage. to the best of our knowledge, our work appears to be the first to leverage administrative claims data, specifically medical codes, for pre-training a large language model to predict clinical outcomes. this approach uniquely utilizes the extensive details available in claims data, filling a notable gap in the current research landscape by applying generative language models in a novel context. ## the proposed framework this section outlines our approach, starting with task definition, followed by our structuring methodology, and concluded with our tokenization process and training criterion. ## conclusions and future work in this work, we have introduced mediclaimgpt, a large language model which has effectively learned the practice of medicine when trained on a massive administrative claims dataset. we showcase its proficiency in the zero-shot prediction of clinical events and downstream classification tasks via various healthcare datasets. its application in creating synthetic claims data, holds tremendous promise for augmenting research and development, as demonstrated by strong evaluation results for fidelity, utility, and privacy. the proficiency of mediclaimgpt's embeddings (discussed in section 4.4.3), suggests that these embeddings can also be effectively utilized for analytical segmentation of patient populations and driving population health management strategy @xcite . additionally, the generative capability of mediclaimgpt in forecasting medical events for patients could lead to new opportunities for digital twins @xcite . for future work, we aim to enrich medi-claimgpt by incorporating a wider range of medical codes, such as laboratory and drug codes, enhancing its medical understanding. additionally, we plan to investigate novel methods for integrat-ing temporal information, like intervals between claims and episodic timeframes, to refine its predictive capabilities. these enhancements will lead to more personalized and efficient care, and expand the strategic application of llms in healthcare. the fourth through sixth characters describe in greater detail the cause, anatomical location and severity of an injury or illness. 3) the seventh character is an extension digit and used to classify an initial, subsequent or sequela (late effect) treatment encounter.