Developing a Clinical Language Model for S wedish: Continued Pretraining of Generic BERT with In-Domain Data (2021)

Abstract: The use of pretrained language models, fine-tuned to perform a specific downstream task, has become widespread in NLP. Using a generic language model in specialized domains may, however, be sub-optimal due to differences in language use and vocabulary. In this paper, it is investigated whether an existing, generic language model for Swedish can be improved for the clinical domain through continued pretraining with clinical text. The generic and domain-specific language models are fine-tuned and evaluated on three representative clinical NLP tasks: (i) identifying protected health information, (ii) assigning ICD-10 diagnosis codes to discharge summaries, and (iii) sentence-level uncertainty prediction. The results show that continued pretraining on in-domain data leads to improved performance on all three downstream tasks, indicating that there is a potential added value of domain-specific language models for clinical NLP.

## introduction pretrained language models, trained on a variety of readily accessible and large-scale unlabeled corpora, and subsequently fine-tuned on downstream tasks using labeled datasets, have led to substantial performance gains across a whole host of nlp tasks. this has contributed to ameliorating a major bottleneck in the development of nlp systems, i.e. the need for access to very large amounts of labeled data for supervised learning. in many cases, obtaining large, task-specific datasets in the form of human-annotated corpora is challenging and prohibitively expensive. as a result, the paradigm of pretraining and fine-tuning has become fundamental for contemporary nlp. in particular, with the introduction of models such as bert @xcite , which are based exclusively on self-attention, i.e. transformers @xcite , and leverage transfer learning techniques, language models have become increasingly accessible; yet, pretraining language models from scratch requires substantial computational resources. while generic language models, trained and released to the public by resource-rich organizations, can be utilized and fine-tuned to perform a particular downstream nlp task without a need for significant resources -neither computational nor in terms of data -it has been shown that their use in specialized domains may be sub-optimal as a result of differences in, for instance, language use and vocabulary @xcite . this has motivated efforts to develop domain-specific language models, e.g. scibert @xcite and biobert @xcite . specialized language models have been developed either by (i) pretraining a language model with in-domain data from scratch, possibly in combination with out-domain data, or by (ii) continuing to pretrain an existing, general language model with in-domain data (domain-adaptive pretraining), either by using large amounts of in-domain data, if available, or by only using task-related unlabeled data (task-adaptive pretraining). the need for language models is particularly pronounced in low-resource settings -both in terms of languages and domains. while there is a publicly available generic language model for swedish, kb-bert @xcite , pretrained using text from the national library of sweden, there is no domain-specific variant for swedish clinical text. in this paper, we report on the development of a clinical language model for swedish. the approach is based on continual pretraining of kb-bert with in-domain data in the form of clinical text. the model is fine-tuned and evaluated on three downstream clinical nlp tasks: (i) detection of protected health information, i.e. a named entity recognition task, (ii) automatic assignment of icd-10 codes to discharge summaries, i.e. a document-level multiclass, multi-label classification task, and (iii) uncertainty classification, i.e. a sentence-level multiclass, single-label classification task. the clinical kb-bert is compared to the original kb-bert and we report downstream performance of various checkpoints during the pretraining process. the domain-specific and generic bert models are further evaluated on a generic ner task in order to understand if performance gains are best explained by the quantity or the domain-specificity of the additional pretraining data. ## related work there has been a substantial amount of effort in recent years dedicated to exploring and developing domain-specific and specialized language models by pretraining with in-domain data, particularly for the biomedical domain and for english. an early and notable effort was the release of biobert @xcite , a bert model pretrained on large-scale biomedical corpora (pubmed abstracts and pmc full-text articles) in addition to general-domain corpora (english wikipedia and bookscorpus). rather than training the model from scratch, biobert was initialized with the general-purpose bert model and also inherited its vocabulary, after which pretraining continued using biomedical data. it was shown that biobert significantly outperforms bert on biomedical nlp tasks. subsequent efforts, e.g. biomegatron @xcite , have shown that additional improvements can be gained by training larger models on even larger in-domain corpora and, in some cases, using a domain-specific vocabulary. in another study, experimental results indicated that training biomedical language models from scratch, as opposed to continued pretraining of a generic language model, may yield improved performance on downstream domain-specific tasks @xcite , although requiring substantial computational resources. domain-specific language models have also been developed for the clinical domain, albeit not for swedish. @xcite pretrained clinical bert models on mimic-iii @xcite using either (i) all types of clinical notes or (ii) discharge summaries only. it was found that initializing the clinical bert models with parameters from biobert, as opposed to parameters from bert, led to better downstream performance, while the types of clinical notes used made little difference on most downstream tasks. however, the clinical language models yielded an increased performance on somebut not all -of the clinical nlp tasks compared to bert and biobert. there have been efforts to develop language models using a combination of biomedical and clinical data. in @xcite , the authors develop such models by applying recent advances in pretraining introduced by roberta @xcite , while studying the impact of using different (combinations of) training corpora and model sizes along with a domain-specific vocabulary. @xcite compare their models with previously published language models on a number of downstream tasks in different domains. their results suggest that using a larger, more powerful generalpurpose language model may be better than using a smaller, less powerful domain-specific language model. however, it is also shown that using indomain data does lead to improved performance: in particular, using clinical data for pretraining leads to large performance gains on clinical tasks but has little impact on biomedical tasks. learning a domain-specific vocabulary yielded improvements on sequence labeling tasks, while the impact was less clear for classification tasks. another very relevant study was conducted by @xcite , where they also explore the potential advantages in continuing to pretrain an existing bert model with in-domain data. the authors explore a number of different settings, such as continuing pretraining on a collection of in-domain corpora for a limited amount of time, continuing the pretraining on the unlabeled training set of the intended downstream task, or continuing pretraining on available unlabeled data directly related to the future downstream task at hand. they explore the duration of each of the continued pretraining setups, and they show that this approach can be very beneficial, especially for the setup in which unlabeled data related to the task at hand is exploited for further pretraining. these results are promising and partly inspired the current work, as all the annotated corpora are very small subsets of the pretraining data. ## methods and data in this paper, we report on the development of a clinical language model for swedish. the hypothesis is that a clinical language model, in this case obtained by continuing to pretrain kb-bert @xcite -a generic language model for swedish -using large amounts of in-domain clinical text, will yield improved performance over a generic language model on downstream clinical nlp tasks. the pretraining process of kb-bert is continued using 17.8 gb of clinical text foot_0 from the research infrastructure health bank 2 -swedish health record research bank at dsv/stockholm university @xcite . this is, in fact, a similar amount of data that was used for pretraining kb-bert. the clinical bert model is pretrained using a geforce rtx 1080 gpu and 17.8 gb of uncompressed clinical text in the form of all available types of clinical notes over a period of seven years. the clinical bert model is trained for one epoch, corresponding to a total duration of ten days. the clinical bert model is fine-tuned and evaluated on three downstream clinical nlp tasks: (i) detection of protected health information, i.e. a named entity recognition task, (ii) automatic assignment of icd-10 codes to discharge summaries, i.e. a document-level multi-class, multi-label classification task, and (iii) uncertainty classification, i.e. a sentence-level multi-class, single-label classification task. the clinical bert model is compared to the original kb-bert and we report downstream performance of various checkpoints during the pretraining process. furthermore, as the clinical bert model is developed using more data compared to kb-bert, it is important to investigate whether potential differences in downstream performance can be attributed to the amount of pretraining data rather than the domain-specificity of the data. to that end, the two language models are also evaluated on a ner task in the general domain. a similar improvement on this task could imply that the clinical bert model is primarily benefiting from additional pretraining data, whereas degraded performance would indicate the value of pretraining specifically on in-domain data. ## results the results indicate that there is a clear benefit in continuing the pretraining process with in-domain data. the clinical kb-bert outperforms its kb-bert counterpart on all three clinical nlp tasks, in terms of f 1 -score. on the phi ner task, it performs close to two percentage points better in terms of f 1 -score, with a large increase in recall and more or less the same precision. the improvement on the icd-10 code assignment task is in the same range as the phi ner task, but in this case yielding a further increase of both precision and recall. on the uncertainty task, the performance improvement is not quite as large as for the other two clinical tasks. however, clinical kb-bert still improves in all the metrics when compared to its kb-bert counterpart. however, on the general-domain ner task, the clinical kb-bert underperforms compared to kb-bert. it falls short by around 4 percentage points in terms of f 1 -score and recall, and by more than 2 percentage points in terms of precision. this indicates that adding more pretraining data does not necessarily lead to better downstream performance, and also that the improved performance on the clinical nlp tasks can likely be attributed to including in-domain data specifically, and not simply more data in general. furthermore, a number of checkpoints during the pretraining process of clinical kb-bert are evaluated on the downstream tasks, the results of which are reported in figure 1 . as can be seen in the figure, for the clinical nlp tasks, there is a positive trend in the performance as the pretraining session progresses. this indicates that, as more data is used, clinical kb-bert becomes better at incorporating and encoding the differences in the distribution of the clinical text and, as a consequence, it becomes better at performing the downstream tasks. however, in the case of the uncertainty multiclass classification task, this trend is not quite as clear: although the vast majority of checkpoints of clinical kb-bert seem to benefit from the continued pretraining with in-domain data, it experiences a low spike towards the end of the epoch, recovering right at the end. in contrast, the performance of clinical kb-bert in the general-domain downstream task seems to follow a steadily degrading trend as the pretraining epoch progresses, and does not show any clear signs of recovering. finally, to illustrate the differences between a general-domain corpus and a clinical-domain corpus, the kb-bert tokenizer is used to process the texts in the phi corpus and the web news corpus, respectively. this tokenizer is a word piece tokenizer, as described by @xcite , and is responsible for constructing the vocabulary -gradually building it from the character level and upwards -by maximizing the likelihood of the training data with respect to the vocabulary. the goal is to investigate how the sentences in the general-domain and clinical-domain corpora are split into tokens, subtokens, and character-level tokens. this is done by calculating the average sentence length, in terms of number of tokens, in the respective corpora when applying the kb-bert tokenizer versus tokenization based on whitespace and regular expressions. as demonstrated in figure 2 , the clinical-domain corpus, after being preprocessed with the kb-bert wordpiece tokenizer, leads to a larger increase in average sentence length compared to the general-domain corpus. ## discussion as demonstrated by the experimental results, there is potentially much to be gained from continuing the pretraining process of an existing generic language model with in-domain data, confirming the findings of previous work. adapting a generic language model to a specific domain by exploiting the availability of unlabeled in-domain data helps bert to better capture the semantics of the target domain as reflected by differences in the underlying distribution. this is highlighted, not only by improved performance on the clinical tasks, but also by the decreased performance on the generaldomain task. these results allow us to better understand the reasons behind the improved performance in the target domain and is an aspect that is often overlooked in similar studies. it indicates that improvements yielded by clinical kb-bert are not solely due to being pretrained on more data -irrespective of domain -but that the domain of the data used for pretraining is indeed an important factor. it is interesting to observe that these improve-ments were yielded by continuing to pretrain the language model for only one epoch, and it is possible that further improvements could be obtained by continuing to pretrain on the in-domain data for several more epochs. moreover, in contrast to similar studies, we also evaluate numerous checkpoints during the pretraining of the the clinical language model. an important observation is that the clinical kb-bert outperforms the original kb-bert on all three clinical nlp tasks after using only 20% of the in-domain data. this indicates that it may be worthwhile to adapt general language models and make them domain-specific even in the absence of enormous amounts of in-domain data. the general domain and the clinical domain differ primarily in the use of different vocabularies. the vocabulary of a news paper article, or a work of literature, follows a different language distribution compared to a clinical note or a discharge summary. clinical texts are typically written in a rather peculiar fashion and contain a large amount of technical terms, as well as (ad-hoc) acronyms and abbreviations, that are not as prevalent and may not even exist in the general domain. there may also be domain-conflicting homonyms, where a word has a completely different meaning in one domain compared to another. due to these differences in vocabulary and frequency, the result of applying a generic language model's tokenizer -in this case that of kb-bert -to a clinical corpus is that the words are likely to be split into subwords, even potentially reaching a character-level split. this was indeed confirmed by the analysis presented in figure 2 . this, in turn, entails that the bert model will use more relevant word-level token representations and more common subword token combinations for the general-domain corpus compared to the clinical-domain corpora, where, instead, there is likely to be a high contribution of subword or even character-level token representations. this impact of the tokenizer in turn implies that the major workload and information encoding falls onto this subset of subword and characterlevel representations during continued pretraining on in-domain data. this not only helps to explain the increased performance on the clinical tasks, but also potentially the performance degradation on the general-domain task since there is a potential mismatch between the representations that are more frequent in the general domain versus the ones that are more frequent -and updated during continued pretraining -in the clinical domain. in future work, this challenge regarding tokenization can be addressed by pretraining a clinical language model from scratch, which would create a tokenizer and vocabulary based on the in-domain clinical data. as shown by previous work, this may lead to further improvements in performance on the clinical tasks. another approach is to manually add specific tokens to the vocabulary of a pretrained model, as explored by @xcite . an informed set of tokens could potentially be extracted by a new tokenizer specifically trained on the in-domain data, and in a later step, incorporate the set difference to the original tokenizer's vocabulary. furthermore, we plan to continue pre-training the current clinical bert model for more epochs in order to investigate whether this will lead to further improvements in performance, as well as training a new model with pseudonymized data with the aim to make this model publicly available. lastly, we also plan to explore and compare different transformer approaches, as well as different pretraining continuation setups, such as using specific parts of the dataset in the spirit of @xcite . these could include more pretraining continuation setups, such as task-specific pretraining, where the unlabeled training set would be used during the pretraining for more epochs. ## conclusions in this paper, we reported on the development of a clinical language model for swedish -the first of its kind. the development of the domain-specific bert model followed the common practice of continuing to train an existing generic language model, kb-bert, with in-domain data. compared to previous efforts to develop clinical language models for english, the model was trained using non-pseudonymized clinical data and, in contrast to previously reported results @xcite , yielded improvements also on the de-identification sub-task of identifying protected health information in clinical text. furthermore, we carefully investigated the effect of further pretraining an existing language model with in-domain data and evaluated a number of checkpoints during the pretraining process on the downstream tasks. the results showed that continued pretraining with in-domain data yielded improvements on the in-domain tasks, but led to worse performance on a general-domain task, indicating that performance gains on the clinical nlp tasks can indeed be attributed to the domainspecificity rather than the sheer size of the additional pretraining data. finally, these results further demonstrate the value of developing domainspecific and specialized language models.