Rare Codes Count: Mining Inter-code Relations for Long-tail Clinical Text Classification (2023)

Abstract: Multi-label clinical text classification, such as automatic ICD coding, has always been a challenging subject in Natural Language Processing, due to its long, domain-specific documents and long-tail distribution over a large label set. Existing methods adopt different model architectures to encode the clinical notes. Whereas without digging out the useful connections between labels, the model presents a huge gap in predicting performances between rare and frequent codes. In this work, we propose a novel method for further mining the helpful relations between different codes via a relation-enhanced code encoder to improve the rare code performance. Starting from the simple code descriptions, the model reaches comparable, even better performances than models with heavy external knowledge. Our proposed method is evaluated on MIMIC-III, a common dataset in the medical domain. It outperforms the previous state-of-art models on both overall metrics and rare code performances. Moreover, the interpretation results further prove the effectiveness of our methods. Our code is publicly available at https://github.com/jiaminchen-1031/Rare-ICD .

## introduction the international classification of diseases (icd) is a worldwide diagnostic tool published and maintained by the world health organization (who). the icd coding, a task of assigning icd codes according to the electronic medical records (emrs), facilitates a lot of activities in health care, such as morbidity and mortality statistical analysis, medical billing and decision support systems @xcite . since the traditional manual emrs coding is time-consuming and prone to error @xcite , its automation has always been attracting attention since 1990s @xcite . most of the existing 1 https://github.com/jiaminchen-1031/rare-icd methods treat the automatic icd coding as a supervised multi-label document classification task @xcite . by learning the text representations with an rnn @xcite , cnn @xcite or transformer @xcite based encoder, the model extracts the code-relevant features via a trainable query matrix and predicts the codes with multiple binary classifiers. ## approach in this section, we introduce the whole architecture of our model, which is illustrated in figure 3 . first, both the medical records and code descriptions are tokenized and embedded via a shared word2vec @xcite . then we adopt a dual encoder architecture to encode code descriptions and medical records respectively. the embedding of code descriptions is put into a relation-enhanced code encoder (section 3.1) to strengthen the connections between codes, especially between the rare codes and frequent codes. we exploit the co-occurrence and the hierarchical structure of icd codes via a series of modules in-side the proposed code encoder. in parallel, the embedding of clinical texts is fed into a clinical text encoder (section 3.2) for contextualization. the outputs of these two encoders interact in the code-text attention block (section 3.3), where the important words are highlighted and combined to generate a new code-specific vector with the representation of each code served as the query. the combination of the weighted words for each code is finally fed into the corresponding binary classifier in the output layer (section 3.4) to calculate the probabilities. ## ablation study we conduct the ablation study concerning the effectiveness of the contextual transformation, code hierarchy and code co-occurrence modules inside the relation-enhanced code encoder. we measure the f1 metrics, which are more sensitive and representative to different models, between our ordinary version and those with a module removed. the results without the entire code encoder, which means using only the embeddings of code descriptions as query, are also listed. as shown in ## conclusions the multi-label clinical text classification is an important task in the domains of both healthcare and natural language processing. in this paper, we reveal the existing problem of traditional methods in capturing the inter-code relations for rare codes. hereby, we propose to strengthen the relations, thus improving the model performance on rare code prediction. we exploit the inter-code relations by encoding code descriptions and incorporating cooccurrence under a code-organ-system hierarchical structure in order to enhance the connections for rare codes. our model is then evaluated on the commonly used mimic-iii dataset and outperforms the other baselines on both rare codes and full codes. the visualisations further demonstrate the advantage of our method on providing more human-understandable explanations. we conduct as well an analysis concerning the design of adjacency matrices and the ablation study to better understand the different components in our method.