Modelling Temporal Document Sequences for Clinical ICD Coding (2023)

Abstract: Past studies on the ICD coding problem focus on predicting clinical codes primarily based on the discharge summary. This covers only a small fraction of the notes generated during each hospital stay and leaves potential for improving performance by analysing all the available clinical notes. We propose a hierarchical transformer architecture that uses text across the entire sequence of clinical notes in each hospital stay for ICD coding, and incorporates embeddings for text metadata such as their position, time, and type of note. While using all clinical notes increases the quantity of data substantially, superconvergence can be used to reduce training costs. We evaluate the model on the MIMIC-III dataset. Our model exceeds the prior state-of-the-art when using only discharge summaries as input, and achieves further performance improvements when all clinical notes are used as input.

## introduction icd @xcite ) coding refers to the task where medical professionals classify clinical diagnoses and medical procedures associated with each patient using standardised taxonomies, which in turn supports billing, service planning and research. the process is manual and laborious in nature ( @xcite , however there is potential to automate it by identifying relevant information from clinical notes, which are already captured in ehr systems. with this in mind, researchers have started to explore whether machine learning models can succeed at this task @xcite . the current research on the icd coding task focuses on the extraction of codes from the discharge summary. this document is commonly written at the end of a hospital stay and provides a textual description of the important diagnoses and procedures for a given patient, making it particularly helpful for the task. however, many other clinical notes are also created during the hospital stay, which can provide important details or useful additional context that may be missing from the discharge summary itself. analysing the full sequence of notes would allow models to make more accurate decisions and make the problem more similar to a real-life setting, where clinicians have to consider all information about a patient for icd coding, rather than information only in a single document. in this work we study how the inclusion of clinical notes across the entire hospital stay can affect performance on the icd coding task. we propose the hierarchical transformers for document sequences (htds) model, which is an adaptation of the hierarchical transformer model @xcite for temporal modelling of document sequences. the model takes text and metadata (such as the time and type of note) from a sequence of multiple documents as input and achieves improved performance when additional clinical notes are used for modelling. we compare different prioritisation criteria for selecting which notes to use as input and how to best represent the sequence information. methods related to superconvergence are applied to speed up the model training process in order to handle the increased size of the data that needs to be processed. our experiments show that the inclusion of additional clinical notes indeed improves model accuracy and leads to better predictions. we evaluate our models against the mimic-iii-50 @xcite test set. when considering only the discharge summaries of each hospital stay as input, our model exceeds the current state-of-the-art performance in terms of micro-f1. when considering all clinical notes as input, further performance improvements across all metrics of interest are observed, exceeding the state-of-the-art performance in micro-f1, micro-auc, macro-auc, and precision@5 scores. ## related work publicly available electronic health record (ehr) datasets, such as the medical information mart for intensive care iii (mimic-iii) dataset @xcite , provide a shared context for researchers to work on icd coding. recent work on icd coding concentrates on the benchmark tasks presented by @xcite , which extracts icd codes from the free-text discharge summary generated at the end of each hospital stay. mullenbach et al. (2018) also publicly release their data preprocessing codes and train/dev/test data splits, and these were followed by later works for comparability of result. in recent years, state-of-the-art work on the icd coding problem commonly used methods based on convolutional neural networks (cnns) or recurrent neural networks (rnns) for text encoding. caml @xcite ) uses a single convolutional layer along with "per-label attention" to extract representations for each label from the convolution output. msattkg @xcite improves the performance further by using a densely connected convolutional network with variable ngram features, and incorporating knowledge graphs to capture relationships between medical codes. ef-fectivecan @xcite uses a deep convolutional approach, with a "squeeze-and-excitation" module that repeatedly compresses and then decompresses the convolutional features. laat @xcite uses a bidirectional lstm to encode the texts, with a per-label attention step on the output to get the final classification. msmn @xcite uses the same architecture as laat, with an additional step of extending code descriptions from the unified medical language system (umls) with synonyms, and using an attention layer with a separate head for each code synonym. researchers using transformer-based models for text encoding experienced difficulties in matching state-of-the-art performance. @xcite apply a range of different transformer-based models but found that none of them outperformed their reimplementation of a simple cnn-based model. @xcite similarly found it difficult to achieve competitive performance and concluded that better methods of handling long input sequences are required to improve the models further. @xcite also find that a simple self-attention network with far less parameters outperformed bert-based models on many tasks. @xcite show that incorporating task-adaptive pre-training, overlapping chunks, and using a large pretrained language model make it possible to achieve performance that is close to, but still slightly below the state-of-the-art. in general, language models that were pretrained on texts in the biomedicine domain, such as clinicalbert @xcite , biobert @xcite , bluebert @xcite , and pubmedbert @xcite tend to achieve higher performance @xcite as compared to language models such as bert @xcite and roberta @xcite which are trained on general domain corpora, as the models have been adapted to the specialised language used in clinical notes. among the range of pretrained language models available for the biomedicine domain, better performance was achieved when a specialised token vocabulary is used @xcite and when the pre-training corpora is closer in nature to those used for the downstream task @xcite . very recently, @xcite identified the restricted capacity of the [cls] token as a potential limiting factor, and showed how using all tokens in the label attention step leads to state-of-the-art performance on the mimic-iii-full problem. however, they do not report results on the mimic-iii-50 problem. while transformer-based language models have been very successful on short sequences of text (bert @xcite and roberta @xcite use a maximum sequence length of 512 tokens), challenges arise when attempting to apply it to longer text sequences due to the quadratic computational complexity of the self-attention mechanism. experiments conducted by @xcite show that transformer models require 3x more processing time compared to cnns, making it more tedious to explore different hyperparameters and modelling strategies. various modifications have been proposed to the transformer architecture to reduce computation costs, in models such as trans-formerxl @xcite , longformer @xcite , and bigbird @xcite , however domain-pretrained models for these architectures are relatively scarce. most transformerbased models for the icd coding problem adapt the hierarchical transformer @xcite , which splits the text into chunks that are encoded separately with the pre-trained language model, and then feeds the output of the [cls] token into a second transformer to allow interaction of information across chunks. to the best of our knowledge, there has been no prior work that attempts to extend the icd coding task with other clinical documents. ## approach our hierarchical transformers for document sequences (htds) model is based on the hierarchical transformer architecture @xcite , with additional adaptations specifically to handle document sequences. figure 1 provides an illustrated diagram of the full htds model architecture. we process documents using the following steps: step 1 -preprocess and chunk: the text in each document is sequentially tokenized and split into chunks, each containing up to t c tokens. every new document or clinical note always starts a new chunk. from these tokenized chunks we select up to n c chunks for processing. if more than n c chunks are available, various prioritisation strategies can be considered to select which chunks to use as model input. in our main model we use a strategy that prioritized diversity in the categories of notes used. to do this, we select the last note by timestamp of each category, and then the second last note of each category, and so on until n c chunks of text are selected. ## conclusion as we work towards automated icd coding, it would be helpful to build models that can consider information that is captured across the patient's ehr record, rather than just the discharge summary (which may not always be exhaustive). such an approach would also be more similar to a reallife setting, where clinicians consider all available information for icd coding, rather than information in a single document. in this paper, we demonstrated the htds model, which is an adaptation of the hierarchical transformer model that considers the text and metadata from the entire clinical document sequence for icd coding. while transformer-based models have faced difficulties achieving competitive performance on the icd coding problem in the past, with htds we show that these challenges can be overcome. when evaluated on the mimic-iii-50 test set using only discharge summaries, htds exceeded the prior state-of-the-art performance in terms of micro-f1 score. when all clinical documents were considered, the performance of htds improved further on all performance metrics of interest, and exceeded prior state-of-the-art performance in all metrics except for macro-f1. the results demonstrate the value of including clinical documents beyond the discharge summary in the icd coding problem. possibilities for improving performance even further are plenty. these include: using a large-sized language model or using overlapping text chunks to reduce fragmentation in the initial encoding step @xcite , considering other transformer architectures for long texts @xcite @xcite , smarter strategies for chunking the input text to reduce fragmentation, further improving the strategy for selecting which text to use as model input (possibly going down to text-level rather than document-level approaches), and incorporating methods to better model rare icd codes @xcite . approaches for improving the computational efficiency and training time of the model can be considered to help to reduce gpu resource requirements, and enable the testing of more models and hyperparameter settings. going even further from here, we could consider multi-modal models that use information across the entire ehr database for icd coding. we hope that our findings will encourage future studies that tap on the full breadth and depth of information available in modern ehr databases today in order to further push the limits of performance on the icd coding problem in future.