On the Contribution of Per- ICD Attention Mechanisms to Classify Health Records in Languages with Fewer Resources than E nglish (2021)

Abstract: We introduce a multi-label text classifier with per-label attention for the classification of Electronic Health Records according to the International Classification of Diseases. We apply the model on two Electronic Health Records datasets with Discharge Summaries in two languages with fewer resources than English, Spanish and Swedish. Our model leverages the BERT Multilingual model (specifically the Wikipedia, as the model have been trained with 104 languages, including Spanish and Swedish, with the largest Wikipedia dumps) to share the language modelling capabilities across the languages. With the per-label attention, the model can compute the relevance of each word from the EHR towards the prediction of each label. For the experimental framework, we apply 157 labels from Chapter XI – Diseases of the Digestive System of the ICD, which makes the attention especially important as the model has to discriminate between similar diseases. 1 https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages

## introduction electronic health records (ehrs) are classified by clinical experts for documentation, reporting global health vital statistics, insurance billing, etc. international classification of diseases (icd) is used world-wide to define diagnostic terms and procedures and serves to encode ehrs. there are thousands of terms encoded within the icd @xcite . for medical experts, reading ehrs, lengthy and technical documents, finding explicit and implicit mentions of diagnoses and procedures for then assigning standard icd codes is cumbersome and requires specific training. in fact, it is well-known that manual encoding is not error-free, as an example, @xcite , estimated that 20% of them were either incorrect or were missing. in this context, natural language understanding brings opportunities to bridge the needs of the society in terms of computer aided coding approaches. in 2006, it was argued that natural language processing (nlp) tools could quickly help identify codes in discharge summaries @xcite . today nlp tools for classifying clinical documents written in english are widespread. even more, languages with scarce resources for biomedical nlp like spanish, italian, swedish, etc., are in the limelight in the last years to develop codification systems as has been done for english. in the context of working towards the codification of documents, in languages with a small number of resources for nlp, different tasks have been addressed. in 2018 @xcite worked with italian, french and hungarian for the automatic codification of death certificates. each death certificate consisted of a few words (on average 20 words) with at least one main diagnosis. in 2020 the codiesp task at clef @xcite consisted on the automatic assignment of icd-10 codes to spanish clinical records with 350 tokens on average. for swedish @xcite the authors mentioned that the corpus was compiled with documents that on average had a length of 96 words. admittedly, multi-label classification is challenging, particularly with extensive label-sets (as it is the case of the icd) and domain-specific corpora, and even more when it comes to dealing with clinical information extraction on languages other than english @xcite . spanish and swedish researchers are striving to bridge this gap, indeed, as the first and relevant step, they gathered corpora conveying patient records @xcite ; @xcite . previous works showed that the multi-label classification problem of ehrs coded with icd-10 can be tackled with an adapted bert architecture @xcite ; @xcite . moreover, we focused just on a sub-set of the icd, i.e. the diseases of the digestive system (the icd codes starting with the letter k). focusing on semantically related diseases poses an added challenge, since the natural language understanding (nlu) in charge of encoding the input ehr must be able to cope with the nuances inherent to the distinction of similar diseases. unarguably, it is easier to distinguish two diseases each belonging to a different body-part than two diseases within the same body-part (as it is this case distinguishing diseases all within the digestive system). in summary, distinguishing semantically different diseases (e.g. gastrointestinal vs cardio-pulmonary) would be easier than distinguishing two diseases within the same speciality. to that end, the lm and the attention mechanisms play the most critical role, so we opted for the transformers models. bert-based approaches have been tested in this context, with attention mechanisms as a strength towards finding relationships between input text with output icd codes. the attention is a mechanism whose effectiveness has also been shown with other architectures such as rnns with lstm units @xcite or convolutional neural networks @xcite . nevertheless, in this context we are dealing with scarce resources and relatively similar codes. in this line, the main scientific contribution of this paper rests on the implementation of a head adapted for bert with multiple label attention mechanisms (instead of a generic one) in order to delve deeper into the nuances of the understanding module. in this work, we have implemented a per-label attention mechanism, and given that regular bert models also have the self-attention mechanism, it allowed us to compare the effect of different attention mechanisms. the per-label attention mechanism allows the model to give a different relevance to each word and icd code pair, contrary to the regular attention mechanism. the experimental results support the approach's acceptable performance, so we decided to release the head for the scientific community. ## corpora we have applied two datasets of languages with scarce resources for this work, i.e., languages with fewer resources than english, specifically, spanish and swedish. both datasets are electronic health records containing discharge summaries from patients. the spanish ehrs are from the emergency services of the basque health public system, conveying records, and therefore labels, from all the medical specialities @xcite . however, the swedish ehrs are only from the gastrosurgery medical specialisation and comes from the research infrastructure health bank -swedish health record research bank foot_1 , at stockholm university. therefore, to have equal label sets, we have selected the icd codes shared between both datasets to carry out the experiments, obtaining 157 codes, all from the chapter xi of the icd-10, i.e., diseases of the digestive system. by selecting the codes of some specialities the number of available ehrs is reduced but the label sets are easier to handle. training specific models on ehrs of specialities improves the performance against training general models @xcite . for the swedish icd-10 corpus data set the swedish kb-bert model @xcite has been applied with good results, see @xcite . here we present a quantitative description and comparison between both datasets. regarding the input, the swedish dataset is more than twice larger in number of ehrs, with 8,909 records in contrast to the 3,891 available spanish ehrs. nevertheless, the vocabulary (i.e., number of unique words) is around three times bigger for the spanish dataset. one explanation is that the spanish ehrs come from several specialities, and therefore there is a higher lexical variability due to the specific terms of each medical specialisation. also, the spanish ehr contains lab tests, which could increase the number of unique words significantly. regarding the output, both datasets are equivalent, with the same set of 157 gastrointestinal icd-10 codes. although this is just a subset of the labels, there are still infrequent codes. for example, only 45 codes from the 157 appear in at least 1% of the ehrs. this fact makes the task even more challenging, as, for around 28% of the labels, there are only a few samples from where the model can learn. even though the number of labels is the same, the distinct label sets (i.e., label combinations that are unique) are larger in the swedish dataset than in the spanish (1,288 and 558, respectively) due to the higher number of records. the ratio between the distinct label sets and the number of records is similar, 6.97 for spanish and 6.91 for swedish, meaning that about the same number of ehrs lead to the same number of unique label sets. the most significant differences come when evaluating the length of the ehrs, as the spanish ehrs are significantly longer. while the spanish records convey 984 words on average, the swedish only have 74 words. the standard deviation is also more prominent in proportion, with 491 for the spanish and 77 for the swedish (note that the standard deviation is higher than the mean). although the records from both datasets are discharge summaries, it seems that not all the swedish records are complete summaries, but instead a summary or even one-sentence synopsis of the patient's outcome. ## methodological approach focusing the attention on the methodology, in @xcite the authors demonstrate the effectiveness of transfer learning with pre-trained language representation model bert without attention for the multi-label classification of german non-technical summaries (ntss) of animal experiments. in e-health 2020 the authors of lópez-garcıa et al. (2020) tackled the task as a multi-label classification problem using bert model @xcite for the automatic clinical coding of medical cases in spanish. nlu results crucial to this task and transformers-based language models (lm) are, doubtlessly, the key strength of most recent approaches such as multi-label biomedical text classification @xcite . all this and the inherent challenges related to our work (e.g. the ability to distinguish concepts leveraging semantically related diseases) motivated us towards bert-based approaches. another fact in favour to this choice rests on the ability to the transfer learning between the two languages and, if possible, get benefits from one language model to the other. that is, the resources from one language can boost the lm of the other one, while the system remains decoupled from the data. in order to tackle the multi-label text classification task, we applied a model with a transformerbased architecture. the problem to solve is the mapping between the input of the ehrs (the raw text, x) and a subset of icds from the entire label set, c, where |c| is the number of codes. the deep learning model is trained for the downstream task with pairs of input and output (i.e., ehr texts and icd codes). the transformer-based neural network model is trained with instances comprising pairs of input (ehr text) and output (icd codes). the j-th instance is described formally as (x j , c j ) ⊆ σ * × {0, 1} |c| . the input-output pair is as follows: x j is the string of any length (comprised by tokens from the vocabulary σ), i.e. the ehr. c j is a presence-bits array. c j i encodes the absence or presence of each code c @xmath0, i.e. the icds assigned to the ehr. from the input text, x j , fed to the language model part of the model, a hidden document representation is obtained. the importance of this rests in that our multi-label classifier is built on top of a bert model (see section 3.1). the lm is the core of the transformer-based nlp models. the principal contribution of this work is the use of the hidden representation to compute attention weights that are label-specific for each input token. after computing the attention, the final output (label predictions) is computed with a fully connected layer that is fed with another document representation got from the label-specific attention layers. to support the reproducible research, we release the code of the per-label attention mechanism with this article. ## experimental framework we propose the following experimental setup to evaluate our bert model's performance with per-icd attention compared to the benchmark (standard bert model) on the multi-label icd classification downstream task. the experimental setup comprises the two minority languages (in terms of in-domain clinical data available), spanish and swedish, and a gastrointestinal label set of 157 labels. each experiment is carried out twice, with the same experimental and training parameters, one with the regular multi-label classification head (as the baseline) and the other with our head with per-icd attention. we show the results from the experimental results in multi-label icd classification is often assessed by means of the area under the roc curve (auc) micro averaging the metric for all the icds involved (denoted as aucm in if we analyse the matrices, it can be observed that the source of improvement of the per-icd model can be broken down; while the true negatives stay close (as with a large label set, the majority of classes are negative), the true positives improves considerably, with an increment of almost 100%. in the same way, the false negatives decrease by around 20%. although the swedish results are in general weaker, this behaviour is appreciated similarly for both languages. therefore, given the results, it seems that our per-icd attention head is able to improve the precision of the regular bert models for icd multi-label classification with large label sets. nevertheless, the per-icd model outperforms regular bert in terms of performance, but also in interpretability capabilities, as it has the ability to export the attention weights, allowing its visualisation. ## discussion within the clinical text mining field, the main weakness tends to be the availability of corpora due to the natural patient's confidentiality policy cohen and demner-fushman (2014). as a result, for the research to make progress, the so important comparability might get compromised. by contrast, through this work the authors are glad to make available their own implementation of the per-icd attention approach foot_2 as a secondary contribution of 0 1 predicted 1 0 true 84.51 15.49 92.24 7.76 arithmetic avg. 0 1 predicted 1 0 true 58.53 41.47 77.91 22.09 weighted avg. (a) baseline bert 0 1 predicted 1 0 true 72.09 27.91 82.45 17.55 arithmetic avg. 0 1 predicted 1 0 true 59.91 40.09 81.34 18.66 weighted avg. (b) per-icd bert figure 3: average heatmaps of swedish models this paper. another aspect related with the corpus is the complexity and length of the input ehr. the average length of the input of the works mentioned névéol et al. (2018b); cappellato et al. (2019) are variable from a few words in the case of italian, hungarian and french to 350 words for the documents written in spanish miranda-escalada et al. @xcite . by contrast, in our paper we deal with documents in spanish and swedish with an average length of 800 (exceeding the aforementioned ones) and 70 respectively. according to these results, the per-label attention mechanism improves precision. while more performance is still necessary for a fully automated system, the results suggest that it is suitable for multi-label classification of ehrs according to the icd standard, specifically applying it as a clinical dss, as the per-icd attention can aid the expert in the ehr codification process. ## conclusions we have dealt with the codification of ehrs of the gastrointestinal service for swedish and spanish hospitals. we have developed a bert model for multi-label classification incorporating a per-label attention mechanism. the results obtained have revealed that the proposed model outperforms the regular bert. we have proved this fact for two languages with minor-ity resources in clinical nlp, showing that solutions of language independent nature work. moreover our proposal generates an interpretable output that helps to know the relevance of the tokens with respect to each icd assigned to the ehr. to sum up, the per-label attention mechanism differentiates semantically icds that are related and aids to explain the core of each label. future work may include testing bert models trained for the specific languages, as the beto model @xcite for spanish.