Intriguing Effect of the Correlation Prior on ICD -9 Code Assignment (2023)

Abstract: The Ninth Revision of the International Classification of Diseases (ICD-9) is a standardized coding system used to classify health conditions. It is used for billing, tracking individual patient conditions, and for epidemiology. The highly detailed and technical nature of the codes and their associated medical conditions make it difficult for humans to accurately record them. Researchers have explored the use of neural networks, particularly language models, for automated ICD-9 code assignment. However, the imbalanced distribution of ICD-9 codes leads to poor performance. One solution is to use domain knowledge to incorporate a useful prior. This paper evaluates the usefulness of the correlation bias: we hypothesize that correlations between ICD-9 codes and other medical codes could help improve language models’ performance. We showed that while the correlation bias worsens the overall performance, the effect on individual class can be negative or positive. Performance on classes that are more imbalanced and less correlated with other codes is more sensitive to incorporating the correlation bias. This suggests that while the correlation bias has potential to improve ICD-9 code assignment in certain cases, the applicability criteria need to be more carefully studied.

## introduction electronic health records (ehrs) contain patient information in the form of clinical notes, structured data tables, and biomedical imaging and time 1 the implementation code is available on github: https: //github.com/nyuolab/text2table series. for easy tracking and analysis of health data across different healthcare systems, and critically for billing purposes, hospitals and insurance companies assign codes of a standardized coding system to characterize the clinical conditions of patients. wrong code assignments may result in billing issues that increase patients' expenses substantially, misdiagnosis, and poor tracking of population level health conditions nationally. the ninth revision of the international classification of diseases (icd-9) is a system used worldwide to classify and code diseases, injuries, and other health conditions. there were extensive efforts studying the automated assignment of icd-9 codes to health records and relevant documents @xcite . with recent developments in nlp, there has been a focus on the use of neural networks @xcite @xcite . one particularly recent direction is in the use of language models. originally introduced in bert @xcite , the recipe of pretraining and finetuning of language models has shown promising performance in many tasks. researchers have applied bert for assigning icd-9 codes from medical documents @xcite @xcite . however, bert and other encoder-based language models perform poorly on icd-9 code assignment @xcite . one challenge is the extremely imbalanced distribution of icd-9 codes. following the distribution of medical conditions in the real world, some codes occur frequently while other codes may appear only once @xcite . it is difficult for models to correctly predict minority codes because few samples exist in the dataset @xcite . a proposed solution is to incorporate domain knowledge that provides useful priors for the minority codes @xcite @xcite . we hypothesize that one useful prior for icd-9 code assignment is the correlation between icd-9 codes and other relevant coding systems. we term other relevant coding systems auxiliary tasks because language models in our experiments predict codes from these systems in addition to icd-9 codes. the auxiliary tasks are current procedural terminology (cpt) codes and diagnosis-related group (drg) codes. this correlation prior stems from the domain knowledge that labels from other coding systems give information about icd-9 codes. for example, patients who underwent artery bypass surgeries (cpt code 33533) are likely to have heart failures (icd-9 code 428.0). to test our hypothesis, we investigate the effect of multitasking on correlated auxiliary tasks and encouraging similar label correlations between training labels and model predictions through regularization. we showed that 1) on average, utilizing correlations hurts language models' performance on predicting icd-9 codes from discharge summaries, 2) for each icd-9 code, utilizing correlations might hurt or help, 3) icd-9 codes that are more imbalanced and less correlated with auxiliary tasks experience larger performance changes (both positive and negative) from incorporating the correlation prior. our findings suggest that the correlation prior has the potential to improve predictions of certain icd-9 codes, but this method suffers from instability when the main task has an imbalanced label distribution and a weak correlation with auxiliary tasks. corrloss corrloss is a regularization technique @xcite ) that encourages consistent label correlations between ground truth and predictions. @xcite uses corrloss on the facial affect recognition task to integrate the correlation priors for facial movements. corrloss can be used in any domain where correlation between prediction targets provides a useful signal. thus, we adopt corrloss to integrate information of the correlations between different kinds of diagnosis and procedure codes. ## methods task overview we formulate the task of code assignment into a multilabel text classification task because each patient has multiple codes corresponding to their discharge summaries. each binary label in the task corresponds to a specific code. formally, our classifier aims to approximate the probability p(y 1 , . . . , y n |x), where each y i is an icd-9 code and x is a discharge summary. the correlation prior we hypothesize that correlations between icd-9 and other coding systems are a useful prior for icd-9 code assignment and choose to incorporate the prior in two ways. first, we added the auxiliary tasks of predicting other medical codes (e.g., cpt). formally, we train a classifier to approximate there are benefits and concerns associated with equation 1, and their trade-off is unclear a priori. one benefit is that extra dependency information from p(z|x, y) could potentially simplify learning p(y, z|x). one drawback is that the additional prediction targets z could worsen the curse of dimensionality. whether the benefit would outweigh the drawback is difficult to determine without running a controlled experiment. second, we used corrloss to encourage similar label correlation patterns between training and predictions. formally, we added a regularization term @xmath0, d j ). each summation term scales with a correlation difference: where dataset we built two datasets from the medical information mart for intensive care iii (mimic-iii) @xcite , a database of ehrs. our first dataset, subsequently referred to as "mimic-iii", contains examples of each patient's discharge summary, and associated diagnosis and procedure codes (diagnosis icd-9, procedure icd-9, cpt, and drg). because this dataset is extremely imbalanced, we further select the top 50 most frequently used codes for each kind of coding system to construct a second dataset that represents a more ideal scenario. following the convention of related literature, we call this dataset "mimic-iii-50" @xcite @xcite . statistics of the mimic-iii dataset are in appendix a. ## experiments to test whether the correlation prior is useful for icd code assignment, we incorporate multitasking (equation 1 ) and corrloss (equation 2 ) into our model and check if they improve performance. specifically, we studied two main tasks (diagno-sis icd-9 codes and procedure icd-9 codes). for each main task, we added one of the three auxiliary tasks: drg codes, cpt codes, and the other icd-9 codes (for diagnosis icd-9 code, the auxiliary task can be procedure icd-9 code, and vice versa). we trained both main-task-only models and multitasking models with and without corrloss. ## results multitasking and corrloss hurt performance on mimic-iii-50 and do not significantly impact performance on mimic-iii. ## analysis since the macro f1 score does not show significant changes from multitasking and corrloss on the full mimic-iii dataset, we investigate whether the performance changes for individual labels. specifically, we analyzed how label imbalance (measured by shannon entropy, defined in appendix d.1) and label correlation (measured by the average absolute pearson correlation coefficient between each main task label and all auxiliary task labels, as defined in appendix d.1) affect the model's performance. for individual icd-9 code, incorporating the correlation prior may hurt or help. figure 1 shows that there exist labels with both negative and positive performance changes. labels that are more imbalanced and less correlated to auxiliary labels experience larger changes. figure 1 shows two relationships: (1) more balanced labels (closer to the right) have less performance changes (spread of dots on the y axis), (2) labels that are more correlated with the auxiliary task (darker dots) have less performance changes (spread along the y axis). all the other plots of different tasks and setups show similar patterns (appendix d.1). in both extreme scenarios (imbalanced label, small correlation with auxiliary labels) and ideal scenarios (balanced labels, high correlation with auxiliary labels), incorporating correlation is more likely to hurt than help. ## discussion since multitasking and corrloss worsen language models' overall performance, it contradicts our hypothesis that the correlations between icd-9 codes and other medical codes would be a useful prior. nevertheless, the performance changes on individual labels are more nuanced and show potential for improving prediction of certain icd-9 codes. we wonder what characterizes the labels that benefit from incorporating the correlation prior (dots with positive changes in figure 1 ). perhaps for those labels, the additional dependency information gained from the auxiliary tasks outweigh the increased learning complexity from a larger output space. a prerequisite for a rigorous investigation would be quantifying the trade-off between the dependency information and the learning complexity. we recognize three limitations that may influence the interpretation of our results and call for future works. first, we did not conduct a hyperparameter search for the regularization strength of corrloss. second, since f1 score decreases are substantial and universal across all experiments on mimic-iii-50, we did not run experiments multiple times with different seeds. third, we did not provide a rigorous explanation of what caused our empirical findings. future works can investigate the plausible hypothesis that the trade-off between the dependency information and the learning complexity causes these findings. besides these limitations, future works can also investigate more scenarios and methods of incorporating the correlation prior.