Automatic ICD Coding via Interactive Shared Representation Networks with Self-distillation Mechanism (2021)

Abstract: The ICD coding task aims at assigning codes of the International Classification of Diseases in clinical notes. Since manual coding is very laborious and prone to errors, many methods have been proposed for the automatic ICD coding task. However, existing works either ignore the long-tail of code frequency or the noisy clinical notes. To address the above issues, we propose an Interactive Shared Representation Network with Self-Distillation Mechanism. Specifically, an interactive shared representation network targets building connections among codes while modeling the co-occurrence, consequently alleviating the long-tail problem. Moreover, to cope with the noisy text issue, we encourage the model to focus on the clinical note’s noteworthy part and extract valuable information through a self-distillation learning mechanism. Experimental results on two MIMIC datasets demonstrate the effectiveness of our method.

## introduction the international classification of diseases (icd) is a healthcare classification system launched by the world health organization. it contains a unique code for each disease, symptom, sign and so on. analyzing clinical data and monitoring health issues would become more convenient with the promotion of icd codes (shull, 2019) @xcite . the icd coding task aims at assigns proper icd codes to a clinical note. it has drawn much attention due to the importance of icd codes. this task is usually undertaken by experienced coders manually. however, the manually process is inclined to be labor-intensive and error-prone @xcite . a knowledgeable coder with medical experience has to read the whole clinical note with thousands of words in medical terms and assigning multiple codes from a large number of candidate codes, such as 15,000 and 60,000 codes in the ninth version (icd-9) and the tenth version (icd-10) of icd taxonomies. on the one hand, medical expert with specialized icd coding skills is hard to train. on the other hand, it is a challenge task even for professional coders, due to the large candidate code set and tedious clinical notes. as statistics, the cost incurred by coding errors and the financial investment spent on improving coding quality are estimated to be $25 billion per year in the us @xcite . automatic icd coding methods @xcite have been proposed to resolve the deficiency of manual annotation, regarding it as a multi-label text classification task. as shown in figure 1 , given a plain clinical text, the model tries to predict all the standardized codes from icd-9. recently, neural networks were introduced @xcite ) @xcite to alleviate the deficiency of manual feature engineering process of traditional machine learning method @xcite in icd coding task, and great progresses have been made. although effective, those methods either ignore the long-tail distribution of the code frequency or not target the noisy text in clinical note. in the following, we will introduce the two characteristics and the reasons why they are critical for the automatic icd coding. long-tail: the long-tail problem is unbalanced data distribution phenomenon. and this problem is particularly noticeable in accompanied by a large target label set. according to our statistics, the proportion of the top 10% high-frequency codes in mimic-iii @xcite occupied 85% of total occurrence. and 22% of the codes have less than two annotated samples. this is intuitive because people usually catch a cold but seldom have cancer. trained with these long-tail data, neural automatic icd coding method would inclined to make wrong predictions with high-frequency codes. fortunately, intrinsic relationships among different diseases could be utilized to mitigate the deficiency caused by long-tail. for example, polyneuropathy in diabetes is a complication of diabetes, with a lower probability than other complications since the long term effect of vessel lesion reflect at nerve would come out in the late-stage. if a model could learn shared information between polyneuropathy in diabetes and more common diseases diabetes, the prediction space would range to a set of complication of diabetes. further, utilizing the dynamic code co-occurrence, (the cascade relationship among complications of diabetes) the confidence of predicting polyneuropathy in diabetes is gradually increased with the occurrence of vessel blockages, angina pectoris, hypertorphy of kidney, respectively. therefore, how to learn shared information with considering dynamic code co-occurrence characteristics, is a crucial and challenging issue. ## method this section describes our interactive shared representation learning mechanism and self-distillation learning paradigm for icd coding. figure 2 shows the architecture of interactive shared representation networks and manifest the inference workflow of our method. we first encode the source clinical note to the hidden state with a multi-scale convolution neural network. then a shared attention module further extracts code relevant information shared among all codes. a multi-layer bidirectional transformer decoder insert between the shared attention representation extraction module and code prediction, establishes connections among shared code relevant representations. ## conclusion this paper proposes an interactive shared representation network and a self-distillation mechanism for the automatic icd coding task, to address the long-tail and noisy text issues. the shared representations can bridge the gap between the learning process of frequent and rare codes. and the code interaction module models the dynamic code cooccurrence characteristic, further improving the performance of tail codes. moreover, to address the noisy text issue, the self-distillation learning mechanism helps the shared representations focus on code-related information in noisy clinical notes. experimental results on two mimic datasets indicate that our proposed model significantly outperforms previous state-of-the-art methods.