Pretrained Biomedical Language Models for Clinical NLP in S panish (2022)

Abstract: This work presents the first large-scale biomedical Spanish language models trained from scratch, using large biomedical corpora consisting of a total of 1.1B tokens and an EHR corpus of 95M tokens. We compared them against general-domain and other domain-specific models for Spanish on three clinical NER tasks. As main results, our models are superior across the NER tasks, rendering them more convenient for clinical NLP applications. Furthermore, our findings indicate that when enough data is available, pre-training from scratch is better than continual pre-training when tested on clinical tasks, raising an exciting research question about which approach is optimal. Our models and fine-tuning scripts are publicly available at HuggingFace and GitHub.

## introduction and background the success of transformer-based models in the general domain @xcite soon encouraged the development of language models for domain-specific scenarios @xcite @xcite @xcite . specifically, in the biomedical domain, there has been a proliferation of models @xcite @xcite since the first biobert @xcite model was published. unfortunately, there is still a significant lack of biomedical and clinical models in languages other than english, despite the increasing efforts of the nlp community @xcite . consequently, general-domain pre-trained language models supporting spanish, such as mbert @xcite and beto @xcite , have been often used as a proxy to build domain-specific systems in the absence of genuine alternatives. for instance, @xcite used mbert and biobert on the pharmaconer @xcite dataset, using a fine-tuning strategy aimed to maximize the results. very recently, new pre-trained clinical language models for spanish have been published @xcite by further pre-training the mbert, beto and xlm-roberta @xcite ) models with a corpus of spanish clinical cases with about 64m tokens. in our work, we go one step further to address the language gap for spanish and train two transformer-based language models from scratch. we employed biomedical and clinical corpora (including clinical texts) gathered by ourselves. we evaluated our models with three different named entity recognition (ner) tasks, since ner constitutes a core task in many clinical nlp scenarios. they obtained significant gains over the general-domain models, and matched or outperformed the domain-specific models in all tasks. ## corpora we built two corpora of very different sizes and nature: an electronic health record (ehr) corpus and a biomedical one. the ehr corpus contains 95m tokens from more than 514k clinical documents (including discharge reports, clinical course notes and x-ray reports). the biomedical corpus includes spanish data from a variety of sources for a total of 1.1b tokens across 2,5m documents, namely: for each biomedical resource, we applied a cleaning pipeline with customized operations designed to read data in different formats, split it into sentences, detect the language, remove noisy and ill-formed sentences, deduplicate and eventually output the data with their original document boundaries. finally, to remove repetitive content, we concatenated the entire corpus and deduplicated it again, obtaining about 1.1b words. these preprocessing steps were applied to all data except the ehr corpus, which was left in its original form. ## models pre-training the models presented in this work were pre-trained from scratch employing a roberta @xcite base model with 12 self-attention layers. following the original training, we only used masked language modeling (mlm) as the pre-training objective with subword masking (swm), as in @xcite . we tokenized the training corpus with the byte-level bpe algorithm @xcite , employed in the original roberta, and learned a vocabulary of 50,262 tokens. we run the training for 48 hours on 16 nvidia v100 gpus of 16gb vram, using adam optimizer (kingma and ba, 2015) with a peak learning rate of 0.0005, 10,000 warm-up steps and an effective batch size of 2,048 sentences. 7 other hyper-parameters were left in their default values as in the original roberta training configuration. training was performed at the document level, preserving document boundaries. 8 we performed a train-validation split based on the number of documents, choosing a total of 2,000 documents for the validation set, corresponding to less than 1% of the entire corpus' documents. we then select the model with the lowest perplexity on the validation set as the best model. we used the corpora described in the previous section to produce two roberta models: a biomedical language model training only with the so-called biomedical resources (bsc-bio-es), 9 and a bio-ehr language model that uses both the biomedical and ehr corpus (bsc-bio-ehr-es). 10 we trained the latter model, the biomedical-ehr, to assess if adding a relatively small ehr data to a large-scale corpora has a positive impact on real-world clinical nlp tasks. ## ner fine-tuning we tested and evaluated our models by fine-tuning the ner task, a key component of information extraction tasks in the clinical domain. indeed, we used it as a testbed to evaluate the effectiveness of our pre-trained models. following the usual finetuning method, employed both for general-domain models @xcite and domain-specific ones @xcite , we added a standard linear layer as a token classification head, and the bio tagging schema @xcite to solve the ner tasks. during fine-tuning, both the pre-trained model and the classification layer's parameters are learned with stochastic gradient descent. we used an adam (kingma and ba, 2015) optimizer and searched for an optimal learning rate out of [8e-6, 1e-5, 2e-5, 3e-5, 5e-5] with linear decay and no warm-up steps. we used a batch size of 32 sequences with a maximum length of 512 tokens and a gradient accumulation of 2 steps, resulting in a total batch size of 64. we trained each configuration using three random seeds. the rest of hyper-parameters were left to the default values of huggingface's codebase @xcite . the complete list of hyper-parameter values is displayed in appendix b. we applied this fine-tuning strategy to three different ner datasets. the first two use annotations on curated medical data (clinical cases extracted from medical literature), whereas the last one uses medical records from the ictusnet project. 11 more details are given below. pharmaconer (gonzalez-agirre et al., 2019) is a track on chemical and drug mention recognition from spanish medical texts. the authors compiled a manually classified collection of clinical case report sections derived from open access spanish medical publications, named the spanish clinical case corpus (spaccc). the corpus contained a total of 1,000 clinical cases and 396,988 words and was manually annotated, with a total of 7,624 entity mentions, corresponding to four different mention types. 12 cantemist (miranda-escalada et al., 2020) is a shared task focused on named entity recognition of tumor morphology, in spanish. the can-temist corpus foot_12 is a collection of 1,301 oncological case reports written in spanish, with a total of 63,016 sentences and 1,093,501 tokens. the ictusnet dataset consists of 1,006 hospital discharge reports of patients admitted for stroke from 18 different spanish hospitals. it contains more than 79,000 annotations for 51 different variables. the dataset is part of the ictusnet project, whose main objective was the development of an information extraction system to support domain experts when identifying relevant information in discharge reports. finally, we remark that our main goal is a headto-head comparison between different models to assess the best model pre-training choice. we were not aiming at maximizing results on the ner tasks and therefore we decided not to use sophisticated classification layers that might improve the performances, such as conditional random field @xcite layers on top of bidirectional long short-term memory recurrent networks @xcite . we argue that a simpler token classification layer better evaluates the quality of model representations than a task-specific layer. unlike @xcite , where authors fine-tuned for 200 epochs (obtaining the best results using 100 epochs), we limit the fine-tuning to 20 epochs, and we do not merge the train and development sets in order to improve the results. we consider that fine-tuning for 200 epochs goes against the pre-training/fine-tuning philosophy that states that fine-tuning should be a relatively inexpensive step @xcite , and also that fine-tuning for less epochs evaluates better the pre-training strategy. ## evaluation and results each fine-tuning was executed on 4 nvidia v100 gpus of 16gb vram. it took around 0. 2 hours to complete the pharmaconer, can-temist and ictusnet tasks, respectively. we then report the overall best scores on the test set, obtained by using the best model's hyperparameters on the development set for each dataset (the standard deviation is computed using all the seeds for that configuration). finally, we also report the models' average scores and standard deviations by computing statistics across all the seeds and the learning rates used for each dataset. the average scores are helpful to indicate which model is more robust to the variation of hyper-parameters, which are the learning rate and initial seed in our case. a higher average score and a smaller standard deviation minimizes the risk of obtaining poor results when performing an extensive hyper-parameter search is not feasible. we compared our models with a generaldomain spanish model (roberta-base-bne) @xcite , a general-domain multilingual model that supports spanish (mbert), a domain-specific english model (biobert), and three domain-specific models based on continual pre-training: mbert-galén (based on mbert), beto-galén (based on beto, a general-domain spanish model), and xlm-r-galén (based on xlm-roberta, a general-domain multilingual model supporting spanish). the results are shown in ## conclusions and future work this work presents the first large-scale biomedical spanish language models trained from scratch, using a large biomedical corpora for a total of 1.1b tokens and an ehr corpus of 95m tokens. we fine-tuned the models on three clinical ner tasks and compared them with both general-domain and other available spanish clinical models. the results show the superiority of our models across the ner tasks, making them competitive candidates for clinical nlp applications. our findings demonstrate the benefits of pre-training from scratch, as seen in @xcite . regarding continual pre-training, the benefits are not clear, especially when continual pre-training is performed with small data, as in the case of the mbert-galén, xlm-r-galén, and beto-galén (note that mbert outperforms mbert-galén in two out of three tasks). our work raises exciting research questions about which pre-training approach is optimal to tackle challenging clinical nlp tasks. we will devote future efforts to address the previous question in detail by providing new models based on continual pre-training and extending our evaluation setting to a diverse range of tasks.