Comparative Analysis of Fine-tuned Deep Learning Language Models for ICD -10 Classification Task for B ulgarian Language (2021)

Abstract: The task of automatic diagnosis encoding into standard medical classifications and ontologies, is of great importance in medicine - both to support the daily tasks of physicians in the preparation and reporting of clinical documentation, and for automatic processing of clinical reports. In this paper we investigate the application and performance of different deep learning transformers for automatic encoding in ICD-10 of clinical texts in Bulgarian. The comparative analysis attempts to find which approach is more efficient to be used for fine-tuning of pretrained BERT family transformer to deal with a specific domain terminology on a rare language as Bulgarian. On the one side are used SlavicBERT and MultiligualBERT, that are pretrained for common vocabulary in Bulgarian, but lack medical terminology. On the other hand in the analysis are used BioBERT, ClinicalBERT, SapBERT, BlueBERT, that are pretrained for medical terminology in English, but lack training for language models in Bulgarian, and more over for vocabulary in Cyrillic. In our research study all BERT models are fine-tuned with additional medical texts in Bulgarian and then applied to the classification task for encoding medical diagnoses in Bulgarian into ICD-10 codes. Big corpora of diagnosis in Bulgarian annotated with ICD-10 codes is used for the classification task. Such an analysis gives a good idea of which of the models would be suitable for tasks of a similar type and domain. The experiments and evaluation results show that both approaches have comparable accuracy.

## introduction the task for automatic encoding of electronic health records (ehr) with standard medical classifications is a hot-topic. the international classification of diseases, 10th revision (icd-10) 1 is one of the most commonly used standard medical classifications due to the availability of translations in several languages. it is a hierarchical classification that encodes each diagnosis into a standard code which is used for statistical analysis and insurance reimbursement. the current solutions for this task are based on a restricted subset of icd-10 codes or address some specific task trained on a small manually annotated corpus. recently some deep learning models like bert @xcite pre-trained transformers were applied for different natural language processing (nlp) and clinical nlp tasks. the first type of transformers is language models that cover common vocabulary on several languages: slavicbert @xcite and multiligualbert @xcite . the second type is transformers that cover specific terminology in english. in this particular case, the base language model is pre-trained with medical terminology by using scientific articles abstracts from pubmed foot_1 or full-text articles from pmc foot_2 : biobert @xcite , clini-calbert @xcite , sapbert @xcite , bluebert @xcite , mt-bert @xcite , pubmedbert @xcite . the later transformers prove that rel-atively high accuracy can be achieved in training for automatic icd-10 classification task for the english language @xcite . we hypothesise that comparable accuracy can be achieved also for languages other than english using either type of pre-trained bert transformers. ## related work the task for automatic icd-10 encoding of textual descriptions of diagnosis was addressed in several research challenges like i2b2 nlp challenges, clef ehealth, etc. the major problem is that icd-10 classification contains more than 11k codes and requires a significant number of labeled training data. in general, there are only available labeled datasets for a limited number of icd-10 codes, which is one of the reasons why this task is not yet solved to the full range of icd-10 codes. lavergne et al presented in @xcite a dataset for for icd-10 coding of death certificates that contains 377,677 labeled statements with 3,457 unique icd-10 codes. usually, the labeled datasets are highly unbalanced that has a huge impact on the annotation method performance. this problem was addressed in @xcite , where the authors apply techniques for imbalance effects reduction, like splitting feature spaces and compressing label dimension. the icd-10 classification task was investigated for several languages. the best performance for languages other than english was achieved with svm models @xcite f1 54.9% for dutch; the longest common subsequence problem @xcite for chinese with f1-score of 81.1%; a hierarchical approach @xcite for chinese with f1 score of 91.08%; information retrieval techniques for turkish @xcite with the best score of 76.5% another approach is to view the problem as a multi-label classification task and use neural networks like cnn, lstm/bilstm, and ha-gru @xcite , or applying bert which has shown good results on this task in german @xcite . hybrid approaches, that combine different models show a slight improvement in the results @xcite . for the bulgarian language was done some preliminary experiments using svm and small training corpora @xcite , where the model achieved f-score 84%. we need to mention that the reported results in this work are based on significantly smaller training and test datasets with lim-ited number of icd-10 classes. in this paper we use big annotated corpora and include almost all icd-10 codes used by medical practitioners in bulgaria. in @xcite we show some successful application of the bert pretrained transformers for icd-10 encoding. inspired by the promising results we will investigate both language models: multiligualbert and slavicbert and will compare them with the state-of-the-art models for medical domain in english: biobert, clinicalbert, bluebert and sapbert. ## deep learning methods for text-based classification bert (bidirectional encoder representations from transformers) @xcite is a deep learning language model pre-trained on a large corpus of data using bidirectional transformers that provides context-aware token and sentence representations. there are multiple bert models for different languages and domains and bert has shown very good results on a variety of different tasks. transfer learning can be applied by using the published models and fine-tuning them with a smaller dataset on the target task. we evaluate multiple bert models by applying additional pre-training for bulgarian medical texts using the masked language task and then finetuning them on the multi-class classification for icd-10 codes. for the masked language task, we mask the standard 15% of tokens and train the model to predict the correct token following the architecture from the original paper @xcite . the goal of training is to minimize the perplexity of the model. we use the language model pre-training dataset to improve bert's understanding of bulgarian medical text. we split the language model pre-training dataset in a proportion of 80:20 -80% for training and 20% for testing. wordpiece is used for tokenization and the original vocabulary from each model is used. as subword tokens are used, all words can be represented with tokens in the vocabulary. to train domain/language-specific extension to the vocabulary, a large corpus of training data would be required which is unavailable for bulgarian. when bert tokenizes the input text, it prepends and appends two special tokens -[cls] and @xcite . the [cls] token pools the information from all tokens in the sentence and in our case represents the diagnosis embedding which we use for text classification. for the multi-class classification task, we use the architecture proposed in @xcite (fig. 2 ). we add a linear layer on top of bert, which uses the [cls] token output from the encoder and is trained to predict the correct icd-10 class using a softmax activation. we return the top 5 classes with the highest probability as a prediction from the classifier as each diagnosis can belong to more than one class. we report accuracy, macro-f1, and mean reciprocal rank (mrr) metrics for the classification task. the multilingual bert model uses bert-base as a starting point and is additionally fine-tuned on the masked language task using wikipedia articles in 104 languages incl. bulgarian foot_14 . biobert is based on bert-base and fine-tuned on pubmed abstracts and pmc full-text articles 16 . bluebert is a model based on bert that is pretrained on pubmed abstracts and (mimic-iii foot_16 ) clinical notes 18 . clinicalbert is initialized from biobert and trained on mimic-iii, which contains around 2 million notes 19 . sapbert is a pubmedbert that was further fine-tuned with synonym pairs from the knowledge base of umls, a collection of biomedical ontologies 20 . slavicbert is a model, derived from mulitlin-gualbert, trained on wikipedia articles in bulgarian, czech and polish and news in russian foot_20 . in the current article, experiments were performed with six different types of bert models. each was additionally fine-tuned on medical articles in bulgarian and then attached to the classification task to associate the diagnosis with the corresponding icd-10 code. clinicalbert is fine-tuned for 20 epochs with a final preplexity of 1.7606 (the 12th epoch was 1.7788). slavicbert is finetuned for 16 epochs and has a perplexity of 5.6693 (the 12th epoch was 5.7312). all other models are trained in 12 epochs. all perplexities can be seen in the ## experiments and results as we can see all the models are doing quite well. what makes the clinicalbert one of the best is that this model is pre-trained on top of many clinical notes, which contain a large amount of medical concepts. many of them are in latin and are the same in their use in different languages. also these notes are most likely quite close to medical diagnoses. it is also the model that has been finetuned for most epochs (20 epochs versus 16 for the here it has best macro f1 and mrr. in order to be able to illustrate in a more understandable way the task we will show three examples of real diagnoses from discharge letters of patients and how two of the classifiers (multilingualbert and biobert) predicted codes of these diagnoses in the in contrast with the results presented in @xcite the slavicbert model in our experiments shows comparable results with the other models, and moreover it is the second ranked model on the basis of accuracy. another difference in our results is that clinicalbert outperforms biobert in all three evaluation metrics -accuracy, macro f1 and mrr. in both cases the reason is longer training (more epochs) for the fine-tuning of the models. ## conclusion in the current article, a comparative analysis of six different bert models is made, each of which is trained on a large amount of data and additionally fine-tuned on a big corpus of medical texts in bulgarian. it can be said that the selected models are a good representative sample for the task on which they are applied, as among them there are models trained on over the top 100 languages, trained on slavic languages, trained on medical and bio literature. the results obtained are quite high and show that all tested models are promising. as future improvements, it would be good for all models to be fine-tuned further, both with more texts and with more epochs. it would be good to equalize the number of epochs of fine-tuning for all models. also other good improvements would be comparing the models before and after fine-tuning and applying cross-validation to more accurately evaluate the models.