D r BERT : A Robust Pre-trained Model in F rench for Biomedical and Clinical domains (2023)

Abstract: In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.

## introduction during the last years, pre-trained language models (plms) have been shown to significantly improve performance on many natural language processing (nlp) tasks. recent models, such as bert @xcite or roberta @xcite , are more and more taking advantage of the huge quantities of unlabeled data thanks to recent unsupervised approaches like masked language models based on the transformers architecture @xcite . most of these plms are frequently pre-trained on general domain corpora, such as news articles, books or encyclopedia. an additional step of fine-tuning can also be applied to use these plms on a targeted task @xcite . although these generic models are used in various contexts, recent works have shown that optimal performance in specialized domains, such as * equal contribution. the adaptation of language models to a domain generally follows two strategies. the first is the training from scratch of a new model using only textual data of the targeted specialty. the second approach, called continual pre-training @xcite , pursues the training of already pre-trained models, allowing them to pass from a generic model to a specialized one. even if studies have shown that the first strategy generally offers better performance @xcite , the second requires a much more constrained number of resources @xcite whether in terms of computing resources or of amount of data. however, domain specific data are generally difficult to obtain, resulting in quite a few specialized plms available. this difficulty is even greater for languages other than english. for the medical domain, data produced during clinical care contain the finesse of medical reasoning and are the most prevalent in terms of quantity. however, they are rarely accessible due to patient privacy constraints. in this paper, we describe and freely disseminate drbert, the first roberta-based plms specialized in the biomedical field in french, and the corpus that had allowed their trainings. we also propose an original study concerning the evaluation of different language model pre-training strategies for the medical field, while comparing it with our model derived from clinical private data, called chubert. in our experiments, plms with publicly available biomedical data can result in similar or better performance compared to highly specialized private data collected from hospital reports or to larger corpora having only generic data. our contributions can be summarized as follows: since its release, it obtains state-of-the-art (sota) performance on almost every nlp tasks, while requiring minimal task-specific architectural modifications. however, the training cost of such a model is very high in terms of computation due to the complexity of each training objective and the quantity of data needed. consequently, new methods emerge and propose more effective ways of performing pre-training. one of them is roberta @xcite . in order to improve the initial bert model, the authors made some simple design changes in its training procedure. they modify the masked-language strategy to perform dynamic masking, remove the next sentence prediction task, increase dramatically the batch sizes and use significantly more data during a longer training period. nowadays, roberta is the standard model for a lot of nlp tasks and languages, including french with camembert model @xcite ). in other languages than english, bert-based models are much rarer and primarily rely on continual pre-training. examples include german @xcite , portuguese @xcite , and swedish @xcite . only the spanish @xcite and turkish (türkmen et al., 2022) models were trained from scratch with biomedical and clinical data from various sources. for french, there is, to our knowledge, no publicly available model specifically built for the biomedical domain. ## pre-training datasets in the biomedical domain, previous works @xcite on plms highlighted the importance of matching the data sources used for its training to the targeted downstream tasks. due to their sensitive nature (protection of user data, protected health information of patients, etc.), medical data are extremely difficult to obtain. massive collection of web data related to this domain appears to be a solution that can overcome this lack. however, these web documents vary in terms of quality. no comparison has been made between plms based on specific domain data from the web and those on private documents from clinical data warehouses, whose quality can be controlled. we extracted two different medical datasets for french. the first one gathers data crawled from a variety of free-of-use online sources, and the second one private hospital stays reports from the nantes university hospital. corpus size #words #sentences nachos large (pub.) 7.4 gb 1.1 b 54.2 m nachos small (pub.) 4 gb 646 m 25.3 m nbdw small (private) 4 gb 655 m 43.1 m nbdw mixed (both) 4+4 gb 1.3 b 68.4 m ## models pre-training in this section, we describe the pre-training modalities of our studied models from two points of view: 1) the influence of the data used (size and nature), and 2) the pre-training strategies of the models. these two levels are respectively detailed in sections 4.1 and 4.2. section 4.3 finally presents the existing state-of-the-art pre-trained models that will be used for comparison purposes. ## downstream evaluation tasks to evaluate the different pre-training configurations of our models, a set of tasks in the medical domain is necessary. while this nlp domain-specific benchmark exists in english (blurb @xcite ), none exist for french. in this section, we describe an original benchmark, summarized in ## results and discussions as previously described, we evaluate the performance of our pre-trained language models proposed for the biomedical domain on a set of public and private nlp downstream tasks related to the medical domain. we first propose to analyze the results according to the different pre-training strategies used (section 6.1) then to focus on the impact of the pre-training data, whether in terms of size or nature (section 6.2). finally, we are interested in the generalization capacities of our domain-specific models by applying and comparing them on general domain nlp tasks (section 6.3). note that all the plms have been fine-tuned in the same way for all downstream tasks and all the reported results are obtained by averaging the scores from four runs. performance on biomedical downstream tasks are reported in tables 6 and 7 for respectively private and public tasks. for readability reasons, the first part of each table presents the existing baseline models results, the second part our specialized models trained from-scratch, and the last part our models using continual pretraining. ## conclusion in this work, we proposed the first biomedical and clinical transformer-based language models, based on roberta architecture, for french language. an extensive evaluation study of these specific models has been performed on an aggregated collection of diverse private and public medical tasks. our open-source drbert models improved the state of the art in all medical tasks against both french general model (camembert) and english medical ones (biobert, pubmedbert and clinical-bert). in addition, we showed that pre-training on constrained resources (4 gb) of web-crawled medical makes it possible to compete with, and even frequently surpass, models trained with specialized data from medical reports. results also highlighted that continual pretraining on an existing domain-specific english model, here pubmedbert, is a more viable solution than on a french domain-generalist model while targeting french biomedical downstream tasks. it needs to further investigate the performance of this approach using more data, similar to what we have done with drbert nachos large . the pre-trained models as well as the pretraining scripts 3 have been publicly released online under a mit open-source license. the main purpose of nachos dataset is to promote the development of robust nlp tools by the community, so, we have decided to make the corpora available for academic research.