Effective Convolutional Attention Network for Multi-label Clinical Document Classification (2021)

Abstract: Multi-label document classification (MLDC) problems can be challenging, especially for long documents with a large label set and a long-tail distribution over labels. In this paper, we present an effective convolutional attention network for the MLDC problem with a focus on medical code prediction from clinical documents. Our innovations are three-fold: (1) we utilize a deep convolution-based encoder with the squeeze-and-excitation networks and residual networks to aggregate the information across the document and learn meaningful document representations that cover different ranges of texts; (2) we explore multi-layer and sum-pooling attention to extract the most informative features from these multi-scale representations; (3) we combine binary cross entropy loss and focal loss to improve performance for rare labels. We focus our evaluation study on MIMIC-III, a widely used dataset in the medical domain. Our models outperform prior work on medical coding and achieve new state-of-the-art results on multiple metrics. We also demonstrate the language independent nature of our approach by applying it to two non-English datasets. Our model outperforms prior best model and a multilingual Transformer model by a substantial margin.

## introduction in multi-label document classification (mldc), we have a set of labeled data {x, y}, @xmath0, where n is the number of documents, d is the feature dimension size of each document and l is the total number of labels. the i th row of y is a multi-hot vector representing the set of labels associated with the i th document. the task is to learn a mapping between x and y so that the labels of each document are predicted correctly. mldc has a great number of practical applications, one of which is automatic medical coding, where a patient encounter containing multiple records are assigned with appropriate medical codes. a large number of medical encounters need to be coded for billing purposes everyday. professional clinical coders often use rule-based or simple ml-based systems to assign billing codes, but the large code space (viz. the icd-10 code system with over 90,000 codes) and long documents are challenging for ml models. in addition, coding requires extracting useful information from specific locations across the entire encounter to support the assigned codes. consequently, effective models with the capability of handling these challenges will have an immense impact in the medical domain by helping to reduce coding cost, improve coding accuracy and increase customer satisfaction. deep learning methods have been demonstrated to produce the state-of-the-art outcomes on benchmark mldc and medical coding tasks @xcite @xcite , but demands remain for more effective and accurate solutions. in this paper, we propose effectivecan, an effective convolution attention network for mldc. our models try to strike a careful balance of simplicity and effective over-parameterization such that we can effectively model long documents and capture nuanced aspects of the whole document texts. such a model is particularly useful for addressing the challenges of automatic medical coding. we evaluate our models on the widely used mimic-iii dataset @xcite , and attain state-of-the-art results across multiple commonly used metrics. we also demonstrate the languageindependent nature of our approach by coding on two non-english datasets. our model outperforms prior best model and a multilingual transformer model by a substantial margin. there is a surge in neural network models employed for automatic medical coding in the past several years. in particular, recent works have utilized attention mechanism to improve automatic coding performance. @xcite applied lstms to produce represtations of the discharge summary and used attention to predict the top 50 codes. @xcite proposed caml that applies separate attention for each label, which generates better label-specific representations for label prediction. they also used the label descriptions to regularize the model (called dl-caml) in an attempt to improve the prediction of rare labels. to improve the classification performance, @xcite used the multi-scale convolutional attention while @xcite employed multi-filter convolution to learn text patterns of different lengths. furthermore, to incorporate the inner relationship of the labels, hypercore @xcite integrated a hyperbolic representation learning method and a graph convolutional network, and @xcite utilized multi-graph knowledge aggregation. @xcite proposed to combine bi-lstm and an extension of structured self-attention mechanism for icd code prediction. ## effectivecan model in this section, we introduce our effectivecan model (figure 1 ), which is composed of four major components: an input layer that transforms the raw document texts into pretrained word embeddings, a deep convolution-based encoder that combines the information of adjacent words and learns meaningful representations of the document texts, an attention component that selects the most important text features and generates label-specific representations for each label, and an output layer that produces the final predictions. the model structure is primarily designed for generating better predictions on multi-label classification tasks from three aspects: (1) generating meaningful representations for input texts; (2) selecting informative features from text representations for label prediction; (3) preventing overconfidence on frequent labels. firstly, in order to obtain high-quality representations of the document texts, we incorporate the squeeze-and-excitation (se) network and the residual network into the convolution-based encoder. the encoder consists of multiple encoding blocks to enlarge the receptive field and capture text patterns with different lengths. secondly, instead of only using the last encoder layer output for attention, we extract all encoding layer outputs and apply the attention to select the most informative features for each label. finally, to cope with the long-tail distribution of the labels, we use a combination of the binary cross entropy loss and focal loss to make the model perform well on both frequent and rare labels. ## results to evaluate the effectiveness of our methods, we compare our model with the existing state-of-theart. the results shown below are generated from the average of five runs with different random seeds for parameter initialization. we also investigate the interpretability of the model. ## ablation study we conducted ablation studies to verify the effectiveness of each module in our model. we compare the results on mimic-iii-full between the ordinary model and the one with a component removed. the results for the macro-and micro-f1 scores are listed in only attending to the first or last res-se layer output leads to worse results. it confirms our argument that the multi-layer attention can capture information from the input at different levels, which further facilitates better predictions. it is also possible to completely remove the attention module, but since @xcite has shown that label-wise attention improves f1, this experiment wasn't deemed informative. compared to the original model, the one without using the focal loss produces a slightly lower result in the micro-f1 but a large reduction in the macro-f1. this verifies the effectiveness of the focal loss in tackling the long-tail distribution of the labels. for the sum-pooling attention model, removing the se module results in the largest performance drop. we have yet to find an explanation for this difference in the two attention models. ## conclusions in this paper, we proposed an effective convolutional attention network for mldc, and showed its effectiveness for medical coding on long documents. our model features a deep and more refined convolutional encoder, consisting of multiple res-se blocks, to capture the multi-scale patterns of the document texts. furthermore, we use the multilayer attention to adaptively select the most relevant features for each label. we employ the focal loss to improve the rare-label prediction without sacrificing the overall performance. our model obtains the state-of-the-art results across several metrics on mimic-iii, and compares favorably with other systems on two non-english datasets.