PLM - ICD : Automatic ICD Coding with Pretrained Language Models (2022)

Abstract: Automatically classifying electronic health records (EHRs) into diagnostic codes has been challenging to the NLP community. State-of-the-art methods treated this problem as a multi-label classification problem and proposed various architectures to model this problem. However, these systems did not leverage the superb performance of pretrained language models, which achieved superb performance on natural language understanding tasks. Prior work has shown that pretrained language models underperformed on this task with the regular fine-tuning scheme. Therefore, this paper aims at analyzing the causes of the underperformance and developing a framework for automatic ICD coding with pretrained language models. We spotted three main issues through the experiments: 1) large label space, 2) long input sequences, and 3) domain mismatch between pretraining and fine-tuning. We propose PLM-ICD, a framework that tackles the challenges with various strategies. The experimental results show that our proposed framework can overcome the challenges and achieves state-of-the-art performance in terms of multiple metrics on the benchmark MIMIC data. Our source code is available at https://github.com/MiuLab/PLM-ICD .

## introduction the clinical notes in electronic health records (ehrs) are written as free-form text by clinicians during patient visits. the notes can be associated with diagnostic codes from the international classification of diseases (icd), which represent diagnostic and procedural information of the visit. the icd codes are a standardized way to encode information systematically and internationally, which could be used for tracking healthcare statistics, quality outcomes, and billing. while icd codes provide several useful applications, manually labelling icd codes has been shown to be very labor-intensive and domain expertise is required @xcite . hence, automatically assigning icd codes to clinical notes has been of broad interest in the medical natural language processing (nlp) community. prior work has identified several challenges of this task, including the large number of labels to be classified, the long input sequence, and the imbalanced label distribution, i.e., the long-tail problem @xcite . these challenges make the task extremely difficult, demonstrating that advanced modeling techniques are required. with the introduction of deep learning models, we have seen tremendous performance improvement on the task of automatic icd coding @xcite @xcite @xcite @xcite @xcite . these methods utilized convolutional neural networks (cnns) @xcite @xcite or recurrent neural networks (rnns) @xcite to transform the long text in clinical notes into hidden representations. state-of-the-art methods employed a label attention mechanism, i.e., performing attention to hidden representations independently for each label, to combat the extremely large label set @xcite . recently, pretrained language models (plms) with the transformer @xcite architecture have become the dominant forces for nlp research, achieving superior performance on numerous natural language understanding tasks @xcite . these models are pretrained on large amount of text with various language modeling objectives, and then fine-tuned on the desired downstream tasks to perform different functionalities such as classification @xcite or text generation @xcite . while plms demonstrate impressive capabili-ties across classification tasks, applying plms to automatic icd coding is still not well-studied. previous work has shown that applying plms to this task is not straightforward @xcite , and the main challenges being: as a result, the performance of plms reported in the prior work is inferior to the state-of-the-art models that did not use pre-trained models by a large margin @xcite . their best model achieved 88.65% in terms of micro-auc, compared with the state-of-the-art 94.9% from the isd model @xcite . this result highlighted that the performance of plms on this task was still far from the conventional models. in this paper, we aim at identifying the challenges met during applying plms to automatic icd coding and developing a framework that could overcome these challenges. we first conduct preliminary experiments to verify and investigate the challenges mentioned above, and then we propose proper mechanisms to tackle each challenge. the proposed mechanisms are: 1) domain-specific pretraining for the domain mismatch problem, 2) segment pooling for the long input sequence problem, and 3) label attention for the large label set problem. by integrating these techniques together, we propose plm-icd, a framework specifically designed for automatic icd coding with plms. the effectiveness of plm-icd is verified through experiments on the benchmark mimic-3 and mimic-2 datasets @xcite . to the best of our knowledge, plm-icd is the first transformer-based pretrained language model that achieves competitive performance on the mimic datasets. we further analyze several factors that affect the performance of plms, including pretraining method, pretraining corpora, vocabulary construction, and optimization schedules. the contributions of this paper are 3-fold: in order to leverage the information of definition of each icd code, rnn and cnn were adopted to encode the diagnostic descriptions of icd codes for better prediction via attention mechanism @xcite . moreover, the prior work tried to consider the hierarchical structure of icd codes @xcite , which proposed a tree-of-sequences lstm to simultaneously capture the hierarchical relationship among codes and the semantics of each code. also, @xcite introduced various ways of leveraging the hierarchical knowledge of icd by adding refined loss functions. recently, @xcite proposed to train icd code embeddings in hyperbolic space to model the hierarchical structure. additionally, they used graph neural network to capture the code co-occurrences. laat @xcite integrated a bidirectional lstm with an improved label-aware attention mechanism. ef-fectivecan @xcite integrated a squeezeand-excitation network and residual connections along with extracting representations from all encoder layers for label attention. the authors also introduced focal loss to tackle the long-tail prediction problem. isd @xcite employed extraction of shared representations among highfrequency and low-frequency codes and a selfdistillation learning mechanism to alleviate the long-tail code distribution. @xcite proposed a framework called read, attend, and code (rac) to effectively predict icd codes, which is the current state-of-the-art model on this task. most recent models focused on developing an effective interaction between note representations and code representations @xcite @xcite . our work, instead, is focusing on the choice of the note encoder, where we apply plms for their superior encoding capabilities. ## challenges for plms in this section, we discuss 3 main challenges for plms to work on automatic icd coding and conduct experiments to verify the severity of the challenges. ## proposed framework the task of icd code prediction is formulated as a multi-label classification problem @xcite . given a clinical note of |d| tokens @xmath0, t 2 , • • • , t |d| } in ehr, the goal is to predict a set of icd codes y ⊆ y, where y denotes the set of all possible codes. typically, the labels are represented as a binary vector @xmath1, 1} |y| , where each bit y i indicates whether the corresponding label is presented in the note. the proposed framework plm-icd is illustrated in figure 1 . the details of the components ## experiments in order to evaluate the effectiveness of our proposed framework, we conduct experiments and compare the results with the prior work. ## analysis this section provides analysis on factors that affect plm's performance on automatic icd coding. ## conclusion in this paper, we identify the main challenges of applying plms on automatic icd coding, including the long text input