M e D a- BERT : A medical D anish pretrained transformer model (2023)

Abstract: This paper introduces a medical Danish BERT-based language model (MeDa-BERT) and medical Danish word embeddings. The word embeddings and MeDa-BERT were pretrained on a new medical Danish corpus consisting of 133M tokens from medical Danish books and text from the internet. The models showed improved performance over general-domain models on medical Danish classification tasks. The medical word embeddings and MeDa-BERT are publicly available.

## introductions large language models (llm) are powerful representation learners and have become the backbone structure of many modern natural language processing (nlp) systems. to learn text representations, llm are first pretrained on a largescale text corpus using self-supervised learning, e.g., masked language modelling. after pretraining, llm are fine-tuned on specific downstream tasks where they have achieved state-of-the-art results on nlp benchmarks such as glue @xcite . however, directly applying these general pretrained models to specialized domains such as the medical have led to unsatisfactory results @xcite . as a solution to this, a second round of in-domain pretraining (domain-adaptive pretraining) has shown to improve the performance of llms that were first trained on a general domain corpus @xcite . domainadaptive pretraining adjusts the weights of the llm to better capture the terminology, style, and nuances that are relevant to the target domain. resource-rich languages such as english have large domain-specific corpuses available that have been used to develop e.g., biomedical @xcite , clinical @xcite , scientific @xcite , and financial @xcite llms that perform better than models trained on general corpuses. these models could potentially be used to improve human decision making, save time, and reduce costs, e.g., by extracting information from scientific articles, identifying potential drug interactions, and helping with nlp tasks such as text classification, named entity recognition, and question answering for each of their specialized domains. for the danish language, only llms trained on a general domain have been made publicly available foot_2 . this paper presents a medical danish bert model (meda-bert)-a llm trained on a new medical danish text corpus. we also used the medical corpus to train medical word embeddings as they still have value in the clinical domain @xcite . to evaluate the medical word embeddings and meda-bert, we used existing medical danish classification datasets. we found that an lstm model using the medical word embeddings outperformed a similar model using general-domain word embeddings, and that meda-bert performed slightly better than a general-domain bert model. ## method this section first describes how the medical corpus was collected and used to pretrain the medical danish word embeddings and meda-bert. next, the datasets used to compare model performances and the fine-tuning procedure is described. ## results ## discussion and limitations this paper presented a new danish medical corpus that was used to train nlp models. the corpus included medical books and text scraped from medical websites that provide information for both citizens and healthcare professionals. we applied different techniques to filter out non-medical data, e.g., by removing documents from non-medical departments or text written by non-healthcare professionals. while these steps did remove a large part of non-medical text, some non-medical text might still be present in the corpus. however, the results showed that models pretrained on the medical corpus performed better than generaldomain models, especially for multiclass classification problems. for the danish language, few medical evaluation datasets are available and therefore the models were only evaluated on classification tasks. moreover, the evaluation datasets were constructed from ehr text which has its own nuances compared to the text of the medical pretraining corpus, e.g., ehr text contains many spelling mistakes whereas the medical corpus contains few grammatical errors. these factors might limit the generalizability of the results. future work should evaluate the models on other tasks, e.g., namedentity recognition and question answering which will provide a better understanding of the models' capabilities. we found continuous small performance improvements by pretraining meda-bert for more epochs. the model might improve with further pretraining but because of limited computational resources and the small rate of improvement, we did not explore this further. the model would also benefit from more medical pretraining data. although this paper presented a large part of the available medical danish text, more data could be collected, e.g., from other medical book publishers and websites. the medical datasets used to evaluate the models are not publicly available because of privacy concerns. for future work, we will strive to publish parts of the medical corpus which requires permission from the text owners. we advise interested researchers to contact us for sharing possibilities. ## conclusion this paper presented a danish medical corpus consisting of 133m tokens. the corpus was used to pretrain medical word embeddings and language models. the models trained on the medical corpus performed better than similar models trained on a general domain.