MDACE : MIMIC Documents Annotated with Code Evidence (2023)

Abstract: We introduce a dataset for evidence/rationale extraction on an extreme multi-label classification task over long medical documents. One such task is Computer-Assisted Coding (CAC) which has improved significantly in recent years, thanks to advances in machine learning technologies. Yet simply predicting a set of final codes for a patient encounter is insufficient as CAC systems are required to provide supporting textual evidence to justify the billing codes. A model able to produce accurate and reliable supporting evidence for each code would be a tremendous benefit. However, a human annotated code evidence corpus is extremely difficult to create because it requires specialized knowledge. In this paper, we introduce MDACE, the first publicly available code evidence dataset, which is built on a subset of the MIMIC-III clinical records. The dataset – annotated by professional medical coders – consists of 302 Inpatient charts with 3,934 evidence spans and 52 Profee charts with 5,563 evidence spans. We implemented several evidence extraction methods based on the EffectiveCAN model (Liu et al., 2021) to establish baseline performance on this dataset. MDACE can be used to evaluate code evidence extraction methods for CAC systems, as well as the accuracy and interpretability of deep learning models for multi-label classification. We believe that the release of MDACE will greatly improve the understanding and application of deep learning technologies for medical coding and document classification.

## introduction in extreme multi-label text classification (xmltc) a document is assigned a small number of labels from an extremely large set of possible labels. this large label space poses a challenge for machine learning (ml) which is compounded by the length of input seen in long-document classification. while there is a wide range of document classification datasets, only a limited number of those contain rationales or evidence associated with the labels. of those that do, none (as of writing) are in the extreme multi-label classification setting or apply to long-documents. we present a new dataset for evidence extraction on long documents in an extreme multi-label classification setting. we also provide benchmark results using established techniques using neural networks. computer-assisted coding (cac) is a real world xmltc application that uses natural language processing (nlp) techniques to extract procedure and diagnosis codes from the documentation of patient encounters. mimic-iii (medical information mart for intensive care) @xcite is an open-access dataset comprised of hospital records associated with patients admitted to the critical care units of the beth israel deaconess medical center. for each patient record/chart, the data related to billing includes diagnostic codes, procedure codes, clinical notes by care providers (discharge summaries, radiology and cardiology reports, nursing notes, etc., all in english), and other patient demographic data. the mimic records were originally coded with the alphanumeric code system icd-9 (international classification of diseases) @xcite , which contains approximately 14,000 codes overall. since the release of mimic-iii, there has been a surge of research on using ml models to predict billing codes based on the clinical text @xcite . however, the mimic database does not contain the association between the billing codes and the clinical notes, i.e., the specific narratives in the notes supporting the codes are not present. cac systems are required to extract text evidence (i.e. rationales) to support the generated billing codes. there is no dataset for reference code evidence as it requires medical coding expertise and is costly to build. as a result, work until this point can only illustrate qualitatively that their models can extract text evidence that looks reasonable to hu-mans. this approach is time-consuming and makes the comparison of different methods extremely difficult. the need for a reference evidence dataset is obvious. in many parts of the world, the icd-9 code system is out of date. most countries are currently using the much more robust and specific alphanumeric code system, @xcite . the u.s. version, icd-10-cm, has approximately 69,000 codes, while the procedures (pcs) have about 82,000 codes. only documents generated as a result of a face-to-face visit with an allowable provider should be reviewed for direct icd-10 code abstraction. this includes progress notes, history and physicals, consults and operatives notes, etc., and excludes nursing notes. for procedure code selection, only a procedure or operative note is acceptable. for these reasons, the ml models trained on the mimic-iii discharge summaries to predict icd-9 codes have limited value for medical coding in reality. mimic-iv @xcite improved upon mimic-iii in many ways, one of which is the addition of icd-10 codes. but at the time of our annotation project, the clinical notes associated with the patient records had not been released. in this paper, we introduce mdace, the first publicly available code evidence dataset foot_0 built on a subset of the mimic-iii clinical records. the dataset contains evidence spans for diagnosis and procedure codes annotated by professional medical coders. each span contains the billing code and the text offsets in the respective clinical note. we provide python scripts for merging our evidence representation with the mimic noteevents table to obtain the true evidence so as to comply with the physionet credentialed health data license. to broaden its use, we automatically map between icd-10 and icd-9 codes so that the evidence can potentially be used with the mimic-iv corpus. mdace addresses a critical need for the automatic evaluation of code evidence generated by cac models as well as the rationales extracted by xmltc systems. ## related work with the recent increased attention to the interpretability of deep learning models, datasets containing explanations in different forms (highlights, free-text, structured) have been curated. @xcite provide a list of 65 datasets for various explainable nlp tasks, @xcite present the results of different explanation generation models trained on these datasets. the primary differences between mdace and existing explanation/evidence/annotator rationale datasets for classification tasks are illustrated in many private datasets have been developed for evidence extraction for medical coding, e.g., @xcite . deyoung et al. ( 2022 ) described a mimic-iii subset annotated with potential evidence spans and assigned a ranked list of icd-10 codes. however, these datasets are not publicly available and cannot be used to improve research on evidence extraction. in addition, mdace was created with an annotation process closely mimicking coding in a professional setting. coders reviewed and annotated charts containing multiple clinical notes instead of individual unrelated notes, and coded both procedure and diagnosis codes. there also exists automatically created datasets, for example, @xcite used a semi-supervised approach to create a silver-standard dataset of clinical codes from only the discharge diagnosis sections of the mimic-iii discharge summary notes, with a small sample validated by humans. there has been a surge in neural network models for automatic medical coding in the past several years. @xcite first introduced a convolutional neural net with an attention mechanism, where the label dependent attention weights were used as token importance measure for the model interpretability. @xcite extended this work by incorporating the squeezeand-excitation network @xcite into the text encoder to obtain better contextual text repre- many of the above works are able to use the attention weights to identify the text snippets that justify code predictions. but there is no quantitative evaluation of the quality of the snippets mostly due to the lack of reference evidence. works that use semi-supervised learning for explanation tasks in nlp more broadly include @xcite al. (2019); pruthi et al. (2020); segal et al. (2020) , where @xcite used a linear tagging model for identifying answer snippets in question answering. although they are not directly related to medical coding, we can apply their approaches for evidence extraction with the help of the mdace dataset. ## challenges and solutions mimic-iii poses a number of challenges for creating a reference code evidence dataset. these challenges include the different coding specialties (inpatient & profee) and code systems (icd-9, icd-10 & cpt). this section discusses these challenges and describes our process to increase the usability of mdace. ## evidence extraction methods this section introduces several evidence extraction methods that we implemented within a convolutional neural network based model to establish baselines for code evidence extraction on mdace. ## experiments and results ## conclusions in this paper, we introduce mdace, the first publicly available code evidence dataset built on a subset of the mimic-iii clinical records. the dataset contains evidence spans for diagnosis and procedure codes annotated by professional medical coders. mdace addresses a critical need for cac research to be able to automatically evaluate the code evidence generated by ml models. to the best of our knowledge, mdace is also the only publicly available dataset with evidence annotations for long documents in an extreme multi-label classification setting. the need for improving the interpretability of text classification models has increased in recent years as they become more complex and opaque. however, datasets with label evidence are rare as the evidence annotations do not occur naturally, nor is the evidence actually used in the real world in those domains, e.g. the rationale annotations on the imdb reviews @xcite . recruiting human subjects, especially domain experts, to create an evidence dataset is an expensive and time consuming process. in addition, many applications require the models to be able to generate local explanations @xcite . mdace is a step toward filling the void and can be used to evaluate and enhance the explainability of dl models. we believe that its release will greatly improve the understanding and application of deep learning technologies for medical coding and text classification. given the recent release of the mimic-iv clinical notes, our next step is to combine the mdace annotations with the mimic-iv dataset and establish baseline performance for icd-10 code prediction and code evidence extraction.