Experimental Evaluation and Development of a Silver-Standard for the MIMIC - III Clinical Coding Dataset (2020)

Abstract: Clinical coding is currently a labour-intensive, error-prone, but a critical administrative process whereby hospital patient episodes are manually assigned codes by qualified staff from large, standardised taxonomic hierarchies of codes. Automating clinical coding has a long history in NLP research and has recently seen novel developments setting new benchmark results. A popular dataset used in this task is MIMIC-III, a large database of clinical free text notes and their associated codes amongst other data. We argue for the reconsideration of the validity MIMIC-III’s assigned codes, as MIMIC-III has not undergone secondary validation. This work presents an open-source, reproducible experimental methodology for assessing the validity of EHR discharge summaries. We exemplify the methodology with MIMIC-III discharge summaries and show the most frequently assigned codes in MIMIC-III are undercoded up to 35%.

## introduction clinical coding is the process of translating statements written by clinicians in natural language to describe a patient's complaint, problem, diagnosis and treatment, into an internationally-recognised coded format @xcite . coding is an integral component of healthcare and provides standardised means for reimbursement, care administration, and for enabling epidemiological studies using electronic health record (ehr) data @xcite . manual clinical coding is a complex, labourintensive, and specialised process. it is also errorprone due to the subtleties and ambiguities common in clinical text and often strict timelines imposed on coding encounters. the annual cost of clinical coding is estimated to be @xmath03 per document, would amount to ∼@xmath1316,000 and stretching the 70 weeks. in this work, we present an experimental evaluation of coding coverage in the mimic-iii discharge summaries. the evaluation uses text extraction rules and a validated biomedical named entity recognition and linking (ner+l) tool, med-cat @xcite to extract icd-9 codes, reconciling them with those already assigned in mimic-iii. the training and experimental setup yield a reproducible open-source procedure for building silver-standard coding datasets from clinical notes. using the approach, we produce a silver-standard dataset for icd-9 coding based on mimic-iii discharge summaries. this paper is structured as follows: section 2 reviews essential background and related work in automated clinical coding, with a particular focus on mimic-iii. section 3 presents our experimental setup and the semi-supervised development of a silver standard dataset of clinical codes derived from unstructured ehr data. the results are presented in section 4, while section 5 discusses the wider impact of the results and future work. ## semi-supervised extraction of clinical codes in this section, we describe the data preprocessing, methodology and experimental design for evaluating the coding quality of mimic-iii discharge sum-maries. we also describe the semi-supervised creation of a silver-standard dataset of clinical codes from unstructured ehr text based on mimic-iii discharge summaries. ## results the following section presents the distribution of manually collected annotations from sampling a np, our validation of updated p a and p na post medcat fine-tuning, and the final distribution of codes found in our produced silver standard dataset. adding annotations to selected text spans directly adds the spans to the medcat dictionary, thereby ensuring further text spans of the same content are annotated by the model -if the text span is unique. we collect 864 annotations after reviewing 4000 randomly sampled dd notes from the a np (assigned, not predicted) dataset. 21.6% of dds provide further annotations suggesting that the majority of missed codes lie outside the dd subsection, or are incorrectly assigned. figure 1 shows the distributions of manually collected code annotations and the current mimic-iii set of clinical codes, grouped by their top-level axis as specified by icd-9-cm hierarchy. we collect proportionally consistent annotations for most groups, including the 390-459 chapter (diseases of the circulatory system), which is the top occurring group in both scales. however, for groups such as 240-279 (endocrine, nutritional and metabolic diseases) and 460-519 (diseases of the respiratory system) we see proportionally fewer manually collected examples despite the high number of occurrence of codes assigned within mimic-iii. we explain this by the dd subsection lacking appropriate detail to assign the specific code. for example codes under 250.* for diabetes mellitus and the various forms of complications are assigned frequently but often lack the appropriate level of detail specifying the type, control status and the manifestation of complication. using the manual amendments made on the 864 new annotations, we re-run the medcat model on the entire dd subsection dataset, producing updated p na, p a and a np datasets. we acknowledge a np likely still includes cases of abbreviations, synonyms as we only subsampled 10 documents per code allowing for further improvements to the model. the medcat fine-tuning process was run until convergence as measured by precision, recall and f1 achieving scores 0.90, 0.92 and 0.91 respectively on a held out a test-set with train/test splits 80/20. the fine-tuning code is made available foot_3 . annotations are available upon request given the appropriate mimic-iii licenses. ## discussion on aggregate, the predicted codes by our medcat model suggest that the discharge diagnosis sections listed in 92% of all discharge notifications are not sufficient for full coding of an episode. unsurprisingly, this confirms that clinicians infrequently document all codeable diagnoses within the discharge summary. although, as previously stated, coders are not permitted to make clinical inferences. therefore, to correctly assign a code, the diagnoses must be present within the documented patient episode within the structured or unstructured data. however, the positive correlation between document length and number of predicted codes indicates that missed codes are more prevalent in highly complex cases with many diagnoses. from a coding workflow perspective, coders operate under strict time schedules and are required to code a minimum number of episodes each day. therefore, it logically follows that the complexity of a case directly correlates to the number of codes missed during routine collection. looking at individual code groups we find 240-279 is not predicted proportionally with assigned codes both in p a and p na. we explain this as follows. firstly, dd subsections generally convey clinically important diagnoses for follow-up care. certain codes such as (250.*) describe diabetes mellitus with specific complications, but the dd subsection will often only describe the diagnoses 'dmi' or 'dmii'. secondly, icu admissions are for individuals with severe illness and therefore are likely to have a high degree of co-morbidity. this is implied by the majority of patients (74%) are assigned between 4 and 16 codes. we also observe e000-e999 and v01-v99 codes are disproportionately not predicted. however, this is expected given that both groups are supplementary codes that describe conditions or factors that contribute to an admission but would likely not be relevant for the dd subsection. in contrast, we observe a disproportionately large number of predictions for 001-139 (infectious and parasitic diseases). this is primarily driven by 0389 (unspecified septicemia). a proportion of these predictions may be in error as the specific form of septicemia is likely described in more detail elsewhere in the note and therefore coded as the more specific form. ## conclusions & future work this work highlighted specific problems with using mimic-iii as a dataset for training and testing an automated clinical coding system that would limit model performance within a real deployment. we identified and deterministically extracted the discharge diagnosis (dd) subsections from discharge summaries. we subsequently trained an ner+l model (medcat) to extract icd-9 codes from the dd subsections, comparing the results across the full set of assigned codes. we find our method covers 47% of all tokens, considering we only take 400 of the ∼7k unique codes and perform minimal data cleaning of the dd subsection. we have shown in section 4.3.2 and 4.3.3 that the medcat predicted codes are proportionally inline with assigned codes in mimic-iii. interestingly, we found a 0.504 positive correlation between dd length and the number of codes predicted by medcat, but not assigned in mimic-iii. this result can be understood by observing that the icu admissions in mimic-iii can be extremely complex, with up to 30 clinical codes assigned to a single episode. the dd subsections alone can contain up to 50 line items indicating highly complex cases where codes could easily be missed. we found that the code group 390-459 (diseases of the circulatory system) is both the most assigned group and the group of codes where there are the most missing predictions from our model. furthermore, codes such as hypertension (4019), sepsis and septicemia (0389, 99591), gastrointestinal hemorrhage (95789), chronic kidney disease (5859), anemia (2859) and chronic obstructive asthma (49320) are all frequently assigned but are also the highest occurring conditions that appear in the dd diagnosis subsection but are not assigned in the mimic-iii dataset. this suggests that mimic-iii exhibits specific cases of undercoding, especially with codes that are frequently occurring in patients but are not likely to be the primary diagnosis for an admission to the icu. as we only use the dd section, there are many codes which likely appear elsewhere in the note that we cannot assign. although 92% of discharge summaries contain dd subsections we only match ∼ 16% of assigned codes. we suggest this is due to: our ner+l model lacking the ability to identify more synonyms and abbreviations for conditions, the dd subsections lacking enough detail to assign codes and in some occasions, little evidence to suggest a code assignment. our textual span coverage, presented in section 4.3.1 demonstrates that we often cover all available discharge diagnosis, although there is still room for improvement as the majority of the coverage distribution is around the 50% mark. for future work we foresee applying the same method to either the entire discharge summary or more specific sections such as 'previous medical history' to surface chronic codeable diagnoses that could be validated against the current assigned code set. researchers would however likely need to address false positive code predictions as clinical coding requires assigned codes to be from current conditions associated with an admission. in conclusion, this work has found that frequently assigned codes in mimic-iii display signs of undercoding up to 35% for some codes. with this finding we urge researchers to continue to develop automated clinical coding systems using mimic-iii, but to also consider using our silver standard dataset or build on our method to further improve the dataset.