Continuous Predictive Modeling of Clinical Notes and ICD Codes in Patient Health Records (2024)

Abstract: Electronic Health Records (EHR) serve as a valuable source of patient information, offering insights into medical histories, treatments, and outcomes. Previous research has developed systems for detecting applicable ICD codes that should be assigned while writing a given EHR document, mainly focusing on discharge summaries written at the end of a hospital stay. In this work, we investigate the potential of predicting these codes for the whole patient stay at different time points during their stay, even before they are officially assigned by clinicians. The development of methods to predict diagnoses and treatments earlier in advance could open opportunities for predictive medicine, such as identifying disease risks sooner, suggesting treatments, and optimizing resource allocation. Our experiments show that predictions regarding final ICD codes can be made already two days after admission and we propose a custom model that improves performance on this early prediction task.

## introduction electronic health records (ehr) are rich repositories of patient information, chronicling their medical history, diagnoses, treatment plans, medications and outcomes @xcite . the aggregation and modeling of this data over time presents a unique opportunity for revealing patterns that can improve patient care, operational efficiency, and healthcare delivery. contained within the ehr are textual notes written by clinicians during patient encounters, which are essential for a comprehensive understanding of patient health. these free-text narratives stand out as a particularly rich source of nuanced information, but their unstructured format and domain-specific language use have left them largely underutilized, compared to more readily available structured data sources @xcite . health records are often accompanied by the international classification of diseases (icd) codesstandardized codes that categorize diagnoses and procedures performed during clinical encounters @xcite . assigning icd codes manually is a highly time-consuming task necessary for billing, therefore previous research has been developing multi-label classification systems to detect applicable codes that should be assigned while writing a given document @xcite . the research focus has been on classifying discharge summaries, which are written at the end of a patient's hospital stay @xcite . while this setup provides a useful proxy task, the complete ehr sequence is much longer, containing detailed reports from nursing and radiology, along with specialized notes on echographies and cardiograms (fcg), among others ( in this work, we investigate the potential of predicting icd codes for the whole patient stay at different time points during their stay. beyond the task of detecting codes for a given note, we treat icd codes as a structured summary of all the treatments provided and diagnoses assigned during a hospital stay. the development of models for predicting this information early in the clinical timeline, based on partial indicators even before these codes have been officially assigned, would open many possibilities for predictive medicine. such systems would go beyond the post-discharge diagnostic practices and could be used for identifying early disease risks, suggesting potential treatments or optimizing hospital resource allocation. we investigate the feasibility of this novel task and evaluate to what extent the final set of icd codes can be predicted at earlier stages during the hospital stay. in addition, we propose a custom model for this task that is able to improve prediction accuracy at different time steps. unlike previous icd code prediction models, the architecture is designed with causal attention to ensure that representations at any point throughout a patient's hospital stay are constructed based on the notes available up to that point, without accessing information in the future. the model is then optimized to predict icd codes after every additional note in the input sequence, instead of only at the discharge summary, teaching it to make predictions at any chosen time point during the hospital stay. this task poses additional challenges, as the length of the complete ehr sequence far exceeds that of the discharge summary and early notes have a weaker correlation with the final labels. we introduce a novel method that both augments the data during training and extends the context during inference, substantially enhancing the performance on early icd code prediction. the code for the model and the experiments are available online. foot_0 ## related work the closest previous research to ours has been on automating icd code assignment. given a document mentioning diagnoses and treatments in freeform text, the aim is to detect the correct codes that should be assigned by the clinician. the first attempts at this task primarily relied on convolutional neural networks (cnns) @xcite @xcite and long short-term memory networks (lstms) @xcite . these models utilized pre-trained word2vec embeddings @xcite and combined neighbouring word representations using convolutional filters or recurrent architectures. despite their simplicity, some of these models achieved very high-performance baselines that were difficult to surpass with transformer approaches @xcite . efforts to apply pre-trained transformers without further modifications to the icd coding problem were unsuccessful @xcite . the discharge summary contains 3,594 tokens on average, while the combined set of notes contains an average of 21,916 tokens per patient stay @xcite . crucial information to predict patient diagnoses is likely to be dispersed throughout these notes, thus models with limited context length risk overlooking a significant portion of relevant data. for this reason, subsequent studies focused on adapting transformer architectures to process longer textual sequences. pubmedbert-hier @xcite employed hierarchical transformers to mitigate the length limitation issue, obtaining substantially better results. this approach segments the document into chunks of 512 tokens and employs a bert-based model pre-trained on the biomedical domain to encode each segment @xcite . the segments are then combined using a hierarchical transformer running over the cls-token embeddings. the trldc model @xcite further improved performance by employing a roberta-based model pretrained from scratch on biomedical articles and clinical notes @xcite . the paat (partition attention) model (kim, 2022) was able to surpass lstm-based models like msmn @xcite on the task of identifying the top 50 labels. paat combines the clinical long-former and a bi-lstm, employing partitionbased label attention for improved performance. hilat (hierarchical label-attention) @xcite achieved strong results on the top 50 labels by utilizing clinicalplusxlnet, which outperforms other transformers like roberta variants, with the downside being that the training speed is four times slower due to its bidirectional context capturing @xcite . the htds (hierarchical transformer for document sequences) model @xcite integrated earlier notes into the input context when making decisions about the discharge summary. this model employs a roberta base transformer and a separate transformer layer running over the individual token representations, not only the cls embeddings. they found that the earlier notes were indeed useful as additional evidence at the end of the hospital stay and provided performance improvements when classifying discharge summaries. all this prior work has trained systems to assign icd codes at the end of the hospital stay, whereas we investigate models for making predictions at any point during the stay. furthermore, while previous work has focused on detecting explicit mentions of diagnoses and treatments in a given text, we investigate to what extent future labels can be inferred based on only earlier documents. ## architecture we investigate a model architecture that can be trained to encode a long temporal sequence of many clinical notes and make predictions at any time point, only using earlier notes as context. the model breaks the sequence into smaller chunks and encodes them using a hierarchical transformer. these chunks are combined with causal label attention, which gathers evidence with label-specific attention heads while ensuring that representations at any time are constructed based on the notes available up to that point, without accessing information in the future. finally, a probability distribution across the labels is predicted at each possible time point. we refer to the model as a label-attentive hierarchical sequence transformer (lahst) and describe it in more detail below. figure 1 provides a diagram of the architecture. ## extending the context hierarchical transformer architectures break long inputs into smaller components and reduce the number of long-distance attention operations, thereby keeping memory and computation requirements more manageable when processing very long sequences. this makes them well-suited for icd code classification, as local context is more important for this task and hierarchical models have been shown to outperform long-context models in this setting @xcite . however, even hierarchical models have difficulty with very long sequences, particularly during training. the gradient must be backpropagated through each individual chunk encoding, which can easily cause memory issues when the models are large and the number of segments exceeds a maximum limit. for this reason, we propose a novel solution for applying hierarchical transformers to very long document sequences, such as the sequences of notes in ehr. we refer to this method as the extended context algorithm (eca). it consists of the following modifications to the training and inference loops. training (algorithm 1). for each episode of training, the loop iterates over the training dataset d train , processing each data sample (s, y), where s is the input sequence and y is the corresponding label. within the loop, a random selection of inference (algorithm 2). during inference, we process all the notes in the sequence in batches of n max chunks. each sequence batch, denoted as s batch , is encoded to obtain embeddings h @xmath0. even if the full sequence does not fit into memory, it can be processed in separate batches to obtain all the h batch embeddings. these embeddings are then concatenated along the batch dimension to obtain chunk embeddings for the complete sequence @xmath1. finally, the collected embeddings are passed through causal attention and masked multi-head label attention to obtain predictions p based on the complete sequence. as the computation can be performed in separate batches and then combined, this allows for considerably longer sequences to be used as input during inference. unlike other methods for extending the context of transformers that rely on reducing or compressing long-distance attention @xcite , this proposed method is also exact -the result is always the same as it would be with a single pass using infinite memory. ## results in addition to the lahst framework described in sections 3 and 4, we also evaluate pubmedbert-hier @xcite and htds @xcite on the early prediction task. htds was trained to consider earlier notes in the context while making decisions about the discharge summary, making it the most likely existing model to also perform well on the early prediction task. in addition, htds results are very close to the state-of-the-art on the mimic-iii dataset, making it a very strong baseline. however, htds is a larger model and requires considerably more gpu resources compared to lahst. therefore, we also report a modified version (htds*) which has a comparable number of parameters. we also include the performance of trldc @xcite from the respective paper as an additional strong baseline on classification of the discharge reports. in ## analyzing the extended context selection of context during inference. the extended context algorithm (eca) allows the model to include much longer ehr sequences in the context during inference (with a generous 181 chunk cap applied in our experiments). we evaluate the effect of this algorithm compared to alternative strategies used in other hierarchical models. the "last" setting uses the most recent 16 chunks of text, illustrating the setting where the sequence is truncated from the beginning in order to fit into the model. the "random" setting samples a random subset of chunks from the sequence instead. the results in ## model interpretability the attention weights in the label-attention layer of lahst can potentially be used as an importance indicator of different input notes. a higher weight is associated with an increased relevance of the particular document to predicting a specific code. for an initial visualization, we average the weights across all the codes to find which document types are most important at different temporal cut-offs. the results are shown in fig. 2 . within the 2-day cut-off, all the reports that have diagnostic characteristics have received the highest attention weights. for example, the echocardiography report is the description of an ultrasound test to identify abnormalities in the heart structure and is used by cardiologists to diagnose heart diseases @xcite . the radiology reports detail the results of imaging procedures such as x-rays and mris to diagnose diseases @xcite . all of such reports are highly technical and are specifically created to assist physicians with diagnostic practices. in the absence of the discharge summary, they are the most valuable document types for making early predictions of icd-9 codes and the network has correctly focused more attention on them. in the "last day" setting, the discharge summary becomes available, containing an overview of the entire hospital stay, and the same model is able to switch most of its attention to it. ## conclusions in this study, we investigated the potential of predicting icd codes for the whole patient stay at different time points during their stay. being able to predict likely diagnoses and treatments in advance would have important applications for predictive medicine, by enabling early diagnosis, suggestions for treatments, and optimization of resource allocation. we designed a specialized architecture (lahst) for this task, which uses a hierarchical structure combined with label attention and causal attention to efficiently make predictions at any possible time points in the ehr sequence. the extended context algorithm was further proposed to allow the model to better handle very long sequences of notes. the system is trained by sampling different sub-sequences of notes, which allows the model to fit into memory while also augmenting the data with variations of available examples. during inference, the whole sequence is then processed separately in batches and combined together with a single attention layer, allowing for lossless representations of very long context to be calculated. our experiments showed that useful predictions regarding the final icd codes for a patient can be made already soon after the hospital admission. the lahst model substantially outperformed existing approaches on the early prediction task, while also achieving competitive results on the standard task of assigning codes to discharge summaries. the model achieved 82.9% auc already 2 days after admission, indicating that it is able to rank and suggest relevant icd codes based on limited information very early into a hospital stay.