C linical M amba: A Generative Clinical Language Model on Longitudinal Clinical Notes (2024)

Abstract: The advancement of natural language processing (NLP) systems in healthcare hinges on language models’ ability to interpret the intricate information contained within clinical notes. This process often requires integrating information from various time points in a patient’s medical history. However, most earlier clinical language models were pretrained with a context length limited to roughly one clinical document. In this study, We introduce ClinicalMamba, a specialized version of the Mamba language model, pretrained on a vast corpus of longitudinal clinical notes to address the unique linguistic characteristics and information processing needs of the medical domain. ClinicalMamba models, with 130 million and 2.8 billion parameters, demonstrate superior performance in modeling clinical language across extended text lengths compared to Mamba and other clinical models based on longformer and Llama. With few-shot learning, ClinicalMamba achieves notable benchmarks in speed and performance, outperforming existing clinical language models and large language models like GPT-4 in longitudinal clinical tasks.

## introduction clinical narratives, such as patient histories, consultation notes, and discharge summaries, contain detailed and complex information that extends over long text sequences @xcite . to fully understand a patient's condition, treatments, and outcomes, nlp systems need to integrate information from various parts of these narratives, which often requires understanding the context provided in those long form text @xcite . understanding the sequence of health events is crucial for diagnoses, treatment plans, and patient monitoring @xcite @xcite . this often involves putting together information from different time points within a patient's health history @xcite . long context enables nlp systems to perform temporal reasoning by tracking events over time longitudinally, which is essential for tasks like predicting disease progression or extracting medical relation @xcite @xcite . it becomes imperative to design models for the need for processing longer texts @xcite . prior studies have introduced mamba @xcite , a selective state space model, that selects and compresses all necessary information into latent space from context, and achieves linear-time efficiency with context length. while these advancements have been primarily directed towards processing general domain text, the unique linguistic features of clinical narratives differ significantly from general domain @xcite , motivating us to develop specialized mamba models in the clinical domain. in this work, we build and publicly release clini-calmamba -a mamba model pretrained on longitudinal clinical notes. furthermore, we demonstrate that clinicalmamba outperforms multiple language models on longitudinal clinical nlp tasks. in particular, our contributions are as follows: ## experiments for each fine-tuning task, we measured the micro precision, micro recall, micro f1 scores, and micro receiver operating characteristic/area under the curve (rocauc). we compared our model with the following baselines: gpt-4 is a large language model designed to understand and generate human-like text based on the input it receives. we applied zero-shot prompting to each downstream task, using acan and original prompt introduced in @xcite . @xcite was accessed securely through the azure openai api. we opt out of human review of the data by signing the additional use case form foot_1 . we set the sampling temperature for decoding to 0.1. asclepius-r @xcite is a clinical generative language model trained on mimic-iii discharge summaries and corresponding instructionanswer pairs. it has 7 billion parameters with a maximum input of 4096 tokens. clinicalllama2 (cllama2) is similar to asclepius-r, but it was trained on all types of mimic-iii note with the same computation budget as clinicalmamba-2.8b (60 hours on 4 a100). it has 7 billion parameters with 4096 max context length. clinicallongformer (clongformer) @xcite is a clinical knowledge enriched version of longformer that was further pretrained using mimic-iii clinical notes. it has 149 million parameters with a maximum input of 4096 tokens. we only used local attention and does not apply global attention for computation efficiency. hierachical clinicalroberta (hi-croberta) @xcite , utilizes multiple embedding from clinical roberta @xcite . it first segment clinical notes into chunks of 512 tokens to obtain their embeddings, embeddings are then pooled by concatenation and finally a linear classification head during downstream task. it has 110 million parameters with a max of 16384 tokens. we did not apply prompt based fine-tuning as this is not a generative model. multirescnn @xcite encode free text with multi-filter residualcnn, and applied label code attention mechanism to enable each icd code to attend different parts of the document. ## results & discussions in this section, we will first compare the model's language modeling ability on mimic-iii clinical notes. we will then describe the evaluation on different clinical information extraction tasks. finally, we will share a case study, which illustrates that our clinicalmamba can recall patient information with long history. clinicalmamba stands as the sole model capable of handling clinical notes of up to 16k tokens. as demonstrated in figure 1 , the perplexity for clinicalmamba-2.8b decreased from 3.11 to 2.61 as the context length expanded from 1k to 16k tokens during inference. this is in contrast to the performance of prior clinical autoregressive language models, where perplexity levels rose with increased context lengths. for instance, with clinicalllama-7b, perplexity escalated from 2.82 to 94.02 as the context length grew from 4k to 16k. this limitation arises because these models were trained on contexts not exceeding 4k, impairing their accuracy for next token prediction when given previous contexts beyond 4k. in the domain of extracting information from longitudinal clinical records, clinicalmamba demonstrates superior performance compared to mamba. clinicalmamba achieved rocauc scores of 91.8, 42.3, and 94.2 on cohort selection, code-rare, and code-common, while mamba obtained rocauc scores of 85.7, 37.8, and 92.8 respectively. clini-calmamba also outperformed previous long-range clinical language models with similar number of parameters. clinicalmamba significantly outperformed hierachical-clinicalroberta and clinical-longformer by relatively 52.7% and 19.1% on rocauc respectively. this is particularly notable in the code-rare task with limited training data (5 shots), where clinicalmamba attained an auc of 91.1, compared to 77.1 of hierarchical-clinicalroberta and 80.5 of clinicallongformer. surprisingly, clinicalmamba-2.8b also outperformed zero-shot gpt-4, achieving f1 scores of 88.8, 56.6, and 73.6 on cohort selection, code-rare, and code-common tasks, whereas gpt-4 obtained a f1 score of 84.8, 33.3, and 68.2 respectively. we also present a case with long history attached in the supplementary material. both clini-calmamba and clinicallongformer models identi- clinicallongformer also misinterpreted the negation in this sentence. in this case, clinicalmamba remembered the correct long-distant sentence that leads to the correct answer, while clinicallongformer preferred wrong short-distant sentence to support its prediction. ## conclusion in this study, we developed and released mamba models pretrained on a large collection of clinical notes. our findings demonstrate the superior performance of our clinicalmamba in extracting information from long text documents compared to other models. we strongly believe that clinical nlp researchers can benefit from such long-context generative language models that alleviates the need of a substantial computational power, without any performance trade-off. building on the groundwork laid by this study, future endeavors can further refine and expand the capabilities of clinicalmamba.