Applications of BERT Models Towards Automation of Clinical Coding in I celandic (2024)

Abstract: This study explores the potential of automating clinical coding in Icelandic, a language with limited digital resources, by leveraging over 25 years of electronic health records (EHR) from the Landspitali University Hospital. Traditionally a manual and error-prone task, clinical coding is essential for patient care, billing, and research. Our research delves into the effectiveness of Transformer-based models in automating this process. We investigate various model training strategies, including continued pretraining and model adaptation, under a constrained computational budget. Our findings reveal that the best-performing model achieves competitive results in both micro and macro F1 scores, with label attention contributing significantly to its success. The study also explores the possibility of training on unlabeled data. Our research provides valuable insights into the possibilities of using NLP for clinical coding in low-resource languages, demonstrating that small countries with unique languages and well-segmented healthcare records can achieve results comparable to those in higher-resourced languages.

## introduction in recent decades, the healthcare industry's transition from paper-based to digital systems, particularly through the adoption of electronic health records (ehr), has opened up new avenues for accessing and utilizing healthcare data in research @xcite . this research has predominantly concentrated on structured data, including diagnostic codes and quantitative data from blood tests and other medical measurements. more recently, there has been a significant shift towards analyzing and using unstructured data, especially with the development of bert-based models. during patient visits and treatments, medical staff compile clinical notes, consisting of unstructured, freeform text. this text often relates to diverse medical codes, like the international classification of diseases (icd) diagnostic codes, encompassing over 70,000 entries. precise icd coding is crucial for the healthcare industry. it plays a vital role in accurately recording patient medical histories, billing for treatments, and enabling research and analysis. nevertheless, traditional clinical coding has been a manual, laborintensive process prone to human error @xcite @xcite . since the transition from paper to digital systems, automating clinical coding has been a key objective. the rapid advancements in natural language processing (nlp), especially with the advent of transformer-based models @xcite , have demonstrated increasing potential in meeting this automation goal for english @xcite . in our study, we investigate this task for icelandic, which is categorized as low to medium resourced language, by using over 10 million ehr notes spanning over 25 years from the landspitali university hospital (luh). previous studies have predominantly focused on english-language datasets, specifically mimic-iii/iv @xcite . for other languages, access to data can be a limiting factor and the datasets studied are usually not large. we review results established across a range of different languages in the literature review section. the primary objective of our study is to ascertain the performance attainable for the icelandic language, through continued pretraining of existing models and various adaptations. this challenge was approached with budgetary constraints in mind, due to the limited computational resources available for conducting on-site studies of sensitive ehr data. we explored various approaches, including continued pre-training on ehr data and conducting an ablation study to compare differences in the fine-tuning step. the highest-performing models attained a micro f1 score of 72.2 and a macro f1 score of 65.2 for the top 100 labels and a micro f1 score of 66.2 and a macro f1 score of 18.3 for the top 8922 labels. we performed an ablation study which revealed that integrating label attention (laat) led to enhanced classification performance @xcite . particularly noteworthy is the fact that the performance mirrors results obtained from higher-resourced languages, underscoring the value of ehrs even for smaller countries with unique languages. due to the sensitive nature of the training data, the release of both the models and the data is not feasible. nevertheless, we are confident that our findings provide valuable insights into the possibilities for languages with limited resources, such as icelandic, in this field. ## discussion this study has demonstrated the feasibility of automating clinical coding in icelandic, a language with limited digital resources, by leveraging a substantial corpus of electronic health records. our findings indicate that transformer-based models, particularly when enhanced with label attention mechanisms, can achieve competitive results in clinical coding tasks. the best-performing model in our study attained micro and macro f1 scores that are not only competitive but also comparable to those achieved in higher-resourced languages. specifically, when measuring performance using auc, our models outperform the english stateof-the-art models. however, we note that due to structural and language differences, these direct comparisons need to be taken with a grain of salt. the implications of our findings are significant for the field of nlp in clinical coding. they suggest that even languages with relatively small digital footprints can benefit from the advances in machine learning and nlp. this is particularly encouraging for smaller countries or those with unique languages, where digital resources may be scarce. our research demonstrates that with the right strategies, such as continued pretraining and model adaptation, it is possible to overcome the challenges posed by limited computational resources. our models, trained on a concise dataset of 100 labels, achieved accuracy comparable to those trained on the english mimic-iii-50 dataset. this trend continued when expanding the label space to 8922 labels, mirroring the performance of models using the full mimic-iii dataset. the success of our models can likely be attributed to the extensive dataset at our disposal and the brevity of icelandic medical records, which may be more conducive to automated coding than the lengthy discharge summaries typically used. this has profound implications for the structure of healthcare data curation moving forward. in fact, recent methods have focused on segmenting clinical notes into sections to facilitate automated clinical coding @xcite . our findings suggest that even languages with fewer resources can reach the performance of well-resourced languages if ample well-segmented data is available. studies in other languages, we face the challenge of inconsistent performance metrics and tasks, as shown in our ablation study focused on the impact of laat and the inclusion of unlabeled or out-ofscope data across datasets of varying sizes. we found that laat @xcite had a marginal impact on smaller datasets but was essential for the expansive 8922-code dataset, likely due to the low frequency of certain codes. intriguingly, training with out-of-scope data led to a slight decrease in f1-scores but an increase in auc scores. an improvement in auc could indicate that the model got better across all decision thresholds, but a decrease in f1-score could mean that it came at the cost of performance on low-frequency label categories. further analysis will need to reveal the correct underlying nature of this tradeoff. our study confirms that additional pre-training on ehr notes using the masked-language objec-tive markedly enhances code classification performance, aligning with prior findings on the benefits of domain-specific pre-training @xcite @xcite . we observed that our furtherpretrained model consistently outperformed ice-bert by a margin of at least 3% in micro f1-score across all dataset sizes, from 31.25k to 1000k, after an equivalent training duration of 1000k steps. however, we also noticed a slight drop in the performance of medicebert when applied to data newer than the pre-training data, suggesting the need for further research on the impact of distribution shifts in medical practice on model performance. our research was conducted under computational resource constraints, which limited the scope of our experiments and underscored the importance of powerful hardware in nlp research, particularly for large datasets. nevertheless, in this resourceconstrained scenario, we found temporal deduplication to be an effective data management strategy, which may be particularly suitable for medical datasets where note repetition is common. looking forward, potential research directions include aggregating all notes from a patient's stay to provide models with a richer context for code classification, contrasting with the current approach of analyzing individual notes. this could address the issue of incomplete code assignments in standalone notes. to manage the resulting long text sequences, techniques like document segment pooling @xcite , various text splitting strategies @xcite , and alternative architectures capable of handling extended contexts, such as mamba @xcite and hyena @xcite , could be explored. in light of these findings and future research directions, we must also consider the ethical implications of automating clinical coding. as we move towards systems that can interpret and categorize medical text, questions of privacy, data security, and the potential for algorithmic bias must be addressed. ensuring that these systems are transparent and equitable will be paramount, particularly as we extend these technologies to a broader range of languages and healthcare systems. moreover, the integration of nlp into clinical workflows raises important considerations about the role of human oversight. the balance between automated efficiency and expert judgment remains a delicate one and given the performance of our models the current aim should be to augment human judgment rather than replace it. ## conclusions in this paper, we presented the results for the first icelandic models trained in the task of multilabel icd-10 classification of ehr notes. we compared their performance on datasets with label spaces of 50-, 100-and 8922-codes, which were accumulated over 25 years at the landspitali university hospital. our models performed similarly to stateof-the-art models trained on the mimic-iii dataset. we explored how training on unlabeled data affected our model performance as well as how label attention impacted the confidence of our models in their choices. we hope that our work provides a foundation and guidance for researchers working in other lowto medium-resource languages to explore the finetuning of pre-trained bert-based models in their respective language on medical text for the task of clinical coding.