A n EMIC : A Framework for Benchmarking ICD Coding Models (2022)

Abstract: Diagnostic coding, or ICD coding, is the task of assigning diagnosis codes defined by the ICD (International Classification of Diseases) standard to patient visits based on clinical notes. The current process of manual ICD coding is time-consuming and often error-prone, which suggests the need for automatic ICD coding. However, despite the long history of automatic ICD coding, there have been no standardized frameworks for benchmarking ICD coding models. We open-source an easy-to-use tool named AnEMIC , which provides a streamlined pipeline for preprocessing, training, and evaluating for automatic ICD coding. We correct errors in preprocessing by existing works, and provide key models and weights trained on the correctly preprocessed datasets. We also provide an interactive demo performing real-time inference from custom inputs, and visualizations drawn from explainable AI to analyze the models. We hope the framework helps move the research of ICD coding forward and helps professionals explore the potential of ICD coding. The framework and the associated code are available here.

## introduction diagnostic coding is the task of assigning alphanumeric codes to diagnoses and procedures after a patient visits a healthcare provider. these codes are typically specified by a medical classification standard called the international classification of diseases (icd). diagnostic coding, or icd coding, is an integral component of medical billing, and integral to claims paid by health insurance carriers. the diagnostic coding process alone accounts for approximately 21% of medical administrative costs in the us @xcite . during this process, a professional coder reviews the patient's medical records, including clinical narratives, and manually selects icd codes. since the task requires in-depth clinical knowledge and understanding of medical records, and importantly, due to the fact that there are a large number of icd codes, the task is laborintensive and error-prone @xcite . these difficulties motivate the need for automatic icd coding systems which perform diagnosis classification given a patient's health record @xcite . this has been the subject of considerable research, with some of the early work dating back to the 1990s @xcite , to more recent deep neural nlp approaches. there are a few outstanding and major challenges in the diagnostic coding task. firstly, the label space, the set of all icd codes, is large, and the label distribution is highly imbalanced. secondly, the input text, i.e., the discharge summaries, is noisy and can contain abstruse medical terms, lesser-known abbreviations, misspelt words, etc. also, they are much longer than what most stateof-the-art models take as input. along with those challenges, the absence of a benchmark has impeded the progress of research. due to privacy restrictions that limit access to even publicly available clinical databases, researchers have to create datasets manually from these, and this results in discrepancies in the actual datasets used in individual papers. for instance, the label set of mimic-iii top-50 dataset varies among the literature, and some of them are even used incorrectly. inconsistency in processing the dataset and the inevitable errors introduced as a result of this makes it hard to compare different methods. in this paper, we introduce a framework for benchmarking automatic icd coding with the mimic clinical database. we name our framework anemic, for an error-reduced mimic icd coding benchmark. to the best of our knowledge, anemic is the first attempt to collate and benchmark different deep learning approaches for automatic icd coding with a configurable pipeline. our contributions can be summarized as follows: the remainder of the paper is organized as follows. in section 2, we discuss popular automatic icd coding approaches and datasets. section 3 details our approaches for preprocessing, training, evaluation, and our demo application. in section 4, we perform a quantitative and qualitative analysis of anemic. finally, we conclude with discussion and future work in section 5. ## icd coding benchmark anemic has been designed so that researchers can easily configure the overall process with config files and therefore, easily start research on icd coding with minimal code. also, the architecture has modularity at the center of its design so that researchers can replace one module with another or with their own implementation. such design enables easy comparison between models and reduces burden while developing new models. figure 1 : the icd coding benchmark pipeline of anemic. we provide a pipeline covering the entire process of icd coding. all steps in the pipeline can be easily configured with yaml files. our system also provides an interactive demo for visualizing model predictions with input attribution scores. this demo will help users analyze the performance and interpretability of their models. in the following subsections, we explain each stage in the pipeline. from now on, we will focus on icd coding dataset from mimic-iii since it is the most widely used dataset for this task. figure 1 illustrates the overall pipeline. ## results in this section, we discuss the quantitative and qualitative results of anemic. on quantitative aspects, we discuss the brief statistics of the datasets and the benchmark results on the our icd coding datasets. for the qualitative results, we present and analyze some example of interpretability visualization from our demo application. ## conclusions and future work in this work, we present anemic, a comprehensive framework for automatic diagnostic coding. it serves as a standardized benchmark for icd coding on mimic-iii by correcting errors in existing datasets and providing popular deep learning-based models. our framework has a modularized and easy-to-use config-based design, and researchers can easily experiment by writing config files or adding custom submodules. we also provide an interactive app for performing real-time inference and visualization for model explainability. anemic is under active development and welcomes contributions from the community. upcoming updates to our pipelines include adding more recent approaches and models, especially those that incorporate additional sources of external knowledge, as well as supporting other datasets like the mimic-ii dataset.