A li BERT : A Pre-trained Language Model for F rench Biomedical Text (2023)

Abstract: Over the past few years, domain specific pretrained language models have been investigated and have shown remarkable achievements in different downstream tasks, especially in biomedical domain. These achievements stem on the well known BERT architecture which uses an attention based self-supervision for context learning of textual documents. However, these domain specific biomedical pretrained language models mainly use English corpora. Therefore, non-English, domain-specific pretrained models remain quite rare, both of these requirements being hard to achieve. In this work, we proposed AliBERT, a biomedical pretrained language model for French and investigated different learning strategies. AliBERT is trained using regularized Unigram based tokenizer trained for this purpose. AliBERT has achieved state of the art F1 and accuracy scores in different down-stream biomedical tasks. Our pretrained model manages to outperform some French non domain-specific models such as CamemBERT and FlauBERT on diverse down-stream tasks, with less pretraining and training time and with much smaller corpora.

## introduction recent contextual language models have achieved tremendous results in almost all domains using textual information. transformers @xcite based pre-trained language models (t-plm) have contributed and continue to contribute to the * the first two authors have equal contribution. the biomedical field is one of the most important domain and its associated textual corpora is one of the fast-growing sources of information in several languages. hence, researchers have leveraged plms to represent biomedical knowledge from different sources, following their success in the general domain. there are plenty of biomedical pre-trained language models (b-plms) that have achieved interesting results and that help decision making in the biomedical field, such as biobert @xcite , pubmedbert @xcite , @xcite , etc. plms are trained using different training mechanisms. the most common are masked language modeling (mlm) @xcite , replaced token detection (rtd) @xcite or next sentence prediction (nsp) @xcite . training a biomedical language model using different strategies does benefit the different downstream tasks. furthermore, b-plms apply various pre-training methods since they borrow some characteristics from already existing plms. commonly used pre-training methods includes continual pre-training (cpt), mixed domain pre-training and domain-specific pre-training (dspt). in this work, dspt was used for training our proposed model from scratch using domain-specific french corpora. furthermore, b-plms use tokens as input. tokenization is indeed the basic step of language model training since it is the tokens that are directly used as discrete input for model pretraining. there are different ways to tokenize a text input. the most common tokenization techniques are byte pair encoding (bpe) (such as; sen-tencepiece, wordpiece, etc.) and unigram subword based tokenization. consideration and implementation of different tokenization techniques are equally important to achieve better performance of b-plms, especially when the model is languagespecific. language-specific plms can use common tokenization techniques like bpe, but they can also tailor the tokenization process and train a tokenizer that can fit a specific language and domain under consideration. in a similar way the biomedical text differs from general domain texts, so the use of custom tokenization allows for better representation of most biomedical vocabulary (words). biomedical language models in languages other than english, i.e. plms that are both domain and language-specific, are quite rare. in the field of non-english language-specific models, there are a few that focus on french language, such as camem-bert @xcite and flaubert @xcite . french is a very rich language and french-based plms @xcite have shown the importance of such languagespecific model for different purposes. however, french biomedical textual information have not been implemented using transformers based plm. yet, there are a few french language word embedding in biomedical domains. @xcite compared different word embedding techniques (word2vec @xcite , glove @xcite ) for french health-related documents. given the disadvantages of embedding words for their representation it is necessary to build b-plms for better representation. in this work, we propose alibert (named after jean-louis-marc alibert the french pioneer of dermatology), a bert-based language-specific and domain-specific biomedical language model. alibert uses a masked language model (mlm) pre-training mechanism which randomly masks some of the tokens from the input biomedical text and predicts the masked tokens based on the context of the input. thereby learning the context of each word according to the biomedical text input. a unigram based tokenizer with a novel regu-larization algorithm has been trained for alibert pre-training. in addition to the mlm, we have also trained an electra-based @xcite ) model called alibert-electra. alibert-electra is trained using the replaced token detection mechanism using the same vocabularies and tokenization steps as alibert. in addition, the lamb optimizer is studied to analyze its computational speed gain during model pre-training. here are the main contributions of our work: this paper is organized in the following manner: first the related work is discussed in section 2, different language-specific and domain-specific plms and their pre-training objectives and strategies are discussed. second, section 3 presents our b-plm alibert with details on architecture, tokenization and optimization. then, section 4 discusses the fine-tuning and evaluation of our models in downstream tasks. next, section 5 explain the experiments and results on the down-stream tasks. then, section 6 discusses the results found and the drawbacks we encountered in detail. finally, section 7 concludes the findings of this paper and points out our future directions concerning the domainspecific and language-specific plms. ## related work in recent years, the number of language models based on transformers @xcite has grown rapidly and their performance has been remarkable in many areas. the pioneers of transformers based plms (t-plms) are bert @xcite and gpt @xcite which are a stack of encoders and decoders of transformers, respectively. consequently, the t-plms can be mainly divided as transformer encoder based models such as albert @xcite , roberta @xcite , electra @xcite , and transformer decoder based model such as bart @xcite , pegasus @xcite , and t5 @xcite . @xcite played an important role for the increase of t-plms and fine-tuning many downsteam tasks. they also paved the way for other languages (other than english), such as @xcite @xcite , to develop language-specific (monolingual) language models. there are very few french language models @xcite @xcite @xcite . camembert @xcite and flaubert @xcite are trained on general knowledge french corpora. camembert used oscar dataset which is composed of 130 gigabytes (gb) of raw french text with 32.7 billion tokens whereas flaubert utilized 71 gb of raw text with 12.7 billion of token. bertweetfr (guo et al., 2021) is another french plm trained on french tweets. bertweetfr is a general domain which is initialized using camembert utilizing the largest french tweet corpora which is composed of 16 gb of 226 million tweets. they took tweets with an average length of 30 tokens. kamal eddine et al. (2021) developed a bart based french language model called barthez which is a generative language model based on bart foot_2 @xcite . barthez used 66 gb (110 gb after tokenization) raw text for pre-training. @xcite investigated the usability of transformer based models for french question answering task and provided a model known as fralbert which is based on the compact language models (parameter efficient bert) called albert @xcite . fral-bert is a compact language model pre-trained on the french version of the wikipedia encyclopedia as of 04/05/2021. their dataset is composed of 4 gb of text and 17 million sentences. there are two french domain-specific plms. the first one is juribert @xcite , it is a french legal language model (language and domain-specific) which is trained on 6.3 gb of raw legal text foot_3 . the second is camembiobert @xcite , it is a fine-tuned camembert @xcite using biomedical text from a french language challenge known as deft ("défi fouille de textes") 5 . @xcite introduced their ongoing work on a clinical french language model, known as eds (entrepôt des données de santé), that uses 21 million french clinical reports from electronic health records (ehr) from several hospitals in the paris area. @xcite claimed that their preliminary results achieved better results than camem-bert @xcite . they have trained eds from scratch (eds-from-scratch) and continuous training over camembert (eds-fine-tuned). regarding domain-specific-language models, @xcite built the first bert based language model in english in the biomedical domain, known as biobert. biobert @xcite ) is built on top of the bert @xcite model using abstracts of biomedical articles. following the publication of biobert, biomedical language models have gained considerable momentum. a survey @xcite studied many publicly available language models in the biomedical domain and provided a survey of systematic literature review, known as ammu. ammu includes 121 articles of biomedical language models. ammu investigated the core b-plms concepts, such as pre-training methods, pre-training tasks, fine-tuning methods and embeddings. furthermore, @xcite disclosed different types of corpora along with the language models that used the corpus. the main corpora included were electronic health record (ehr), radiology reports, social media texts and scientific literature. they have listed out the most common learning objectives such as masked language modeling (mlm), replaced token detection (rtd), next sentence prediction (nsp), sentence order prediction (sop) and span boundary objective (sbo). there are few non-english transformer-based biomedical plms @xcite @xcite . most of the models are pre-trained using the continual pretraining (cpt) approach which means they used already pre-trained language-specific general knowledge ptm. we invite readers to refer to the ammu survey for details @xcite . to the best of our knowledge, there is not yet a french biomedical transformer based plm trained from scratch. however, as mentioned above we are aware of an ongoing work on a plm for french clinical reports using proprietary ehr @xcite from the literature we can clearly see that there is a gap in pre-trained language models for french biomedical text mining. hence, our primary goal is to address this gap and enhance the tokenization of french biomedical texts. instead of relying solely on general tokenization methods, we have standardized the tokenization process specifically for french biomedical texts. ## methods this sections focuses on how the proposed pretrained language model, alibert, was built. it describes the pre-training strategy and architecture, pre-training corpora, tokenization and optimization of our models. ## fine-tuning and model evaluation in order to evaluate the level of understanding of french biomedical tasks by alibert, we have finetuned alibert on standard pre-trained language model evaluation tasks such as biomedical named entity recognition (ner), biomedical text classification, etc. below, we discussed how the tasks are trained. ## experiments and results alibert base was trained on 48 gpus nvidia a100 (12 nodes each with 4 gpus) for 20 hours with 512 input tokens and a batch size of 960 (20 batch size for each gpu). we have used a vocabulary of 40k sub-word units which are built using unigram tokenization algorithm. our models have been evaluated using the abovementioned fine-tuning models and on the masked token prediction. the results found using our models have been compared to the camembert (martin et al., 2020) french plm which is the state-ofthe-art in french language. unfortunately, we were not able to compare our models with biomedical plms due to the lack of french plm in biomedical domain. the downstream tasks on which our models have been evaluated are biomedical ner, classification and masking language modeling (mlm). the downstream datasets are not included in the training dataset of our models. however, there might be an overlap with the quaero-medline article titles. the results obtained on these tasks are detailed below. ## discussion our pre-trained language models trained on indomain (biomedical) textual documents tend to outperform models that are trained on general domain textual documents which is also seen on the literature review of pre-trained language models for english language such as biobert @xcite , pubmedbert @xcite , etc. training plms using the masked language model (mlm) objective shows somewhat better results, but the difference is not significant compared to the replaced token prediction (mlm) objective. moreover, choosing the right optimizer like lamb has an effect on the training speed of the pre-trained models but not on the performance of the models. during the training of our models different types of tokenizers, such as, unigram, wordpiece, senten-cepiece, bpe, etc. are trained and compared with each other. in addition, to improve the models, continu-ous training on a general purpose pre-trained language, such as camembert, could be implemented. since our tokenizers were a bit different and our goal is to study a purely biomedical plm, we have not investigated it yet. ## conclusion this paper proposes a french biomedical pretrained language model that was trained on several corpora of french biomedical textual materials. two variants of the model are proposed using two different pre-training strategies. alibert is a pre-trained model based on bert @xcite which used the pre-training strategy of masking language models (mlm). alibert-electra is based on electra @xcite and used a replaced token prediction (rtp) learning strategy. furthermore, a tokenization adaptation strategy was introduced as a building block for pre-training the two proposed models. a lamb optimizer has also been tested to speed-up the learning of alibert. the proposed pre-training models have been tested on different downstream tasks and achieved state-of-the-art results on different tasks. biomedical entity recognition (ner) and biomedical text classification downstream tasks are fine-tuned using different biomedical textual documents. hence, alibert is expected to be used by different organization and practitioners that work with biomedical text for better understanding and to help make informed decisions regarding biomedical situations.