m DAPT : Multilingual Domain Adaptive Pretraining in a Single Model (2021)

Abstract: Domain adaptive pretraining, i.e. the continued unsupervised pretraining of a language model on domain-specific text, improves the modelling of text for downstream tasks within the domain. Numerous real-world applications are based on domain-specific text, e.g. working with financial or biomedical documents, and these applications often need to support multiple languages. However, large-scale domain-specific multilingual pretraining data for such scenarios can be difficult to obtain, due to regulations, legislation, or simply a lack of language- and domain-specific text. One solution is to train a single multilingual model, taking advantage of the data available in as many languages as possible. In this work, we explore the benefits of domain adaptive pretraining with a focus on adapting to multiple languages within a specific domain. We propose different techniques to compose pretraining corpora that enable a language model to both become domain-specific and multilingual. Evaluation on nine domain-specific datasets—for biomedical named entity recognition and financial sentence classification—covering seven different languages show that a single multilingual domain-specific model can outperform the general multilingual model, and performs close to its monolingual counterpart. This finding holds across two different pretraining methods, adapter-based pretraining and full model pretraining.

## introduction the unsupervised pretraining of language models on unlabelled text has proven useful to many natural language processing tasks. the success of this approach is a combination of deep neural networks @xcite , the masked language modeling objective @xcite , and large-scale corpora @xcite . in fact, unlabelled data is so important that better downstream task performance can be realized by pretraining models on more unique tokens, without repeating any examples, instead of iterating over smaller datasets @xcite . when it is not possible to find vast amounts of unlabelled text, a better option is to continue pretraining a model on domainspecific unlabelled text @xcite , referred to as domain adaptive pretraining @xcite . this results in a better initialization for consequent fine-tuning for a downstream task in the specific domain, either on target domain data directly @xcite , or if unavailable on source domain data @xcite . the majority of domain-adapted models are trained on english domain-specific text, given the availability of english language data. however, many real-world applications, such as working with financial documents @xcite , biomedical text @xcite , and legal opinions and rulings @xcite , should be expected to work in multiple languages. for such applications, annotated target task datasets might be available, but we lack a good pretrained model that we can finetune on these datasets. in this paper, we propose a method for domain adaptive pretraining of a single domain-specific multilingual language model that can be fine-tuned for tasks within that domain in multiple languages. there are several reasons for wanting to train a single model: (i) data availability: we cannot always find domain-specific text in multiple languages so we should exploit the available resources for effective transfer learning @xcite . (ii) compute intensity: it is environmentally unfriendly to domain-adaptive pretrain one model per language @xcite , and biobert was domain adaptive pretrained for 23 days on 8×nvidia v100 gpus. (iii) ease of use: a single multilingual model eases deployment when an organization needs to work with multiple languages on a regular basis @xcite . our method, multilingual domain adaptive pretraining (mdapt), extends domain adaptive pretraining to a multilingual scenario, with the goal of training a single multilingual model that performs, as close as possible, to n language-specific models. mdapt starts with a base model, i.e. a pretrained multilingual language model, such as mbert @xcite or xlm-r @xcite . as monolingual models have the advantage of language-specificity over multilingual models @xcite , we consider monolingual models as upper baseline to our approach. we assume the availability of english-language domain-specific unlabelled text, and, where possible, multilingual domain-specific text. however, given that multilingual domainspecific text can be a limited resource, we look to wikipedia for general-domain multilingual text @xcite . the base model is domain adaptive pretrained on the combination of the domain-specific text, and general-domain multilingual text. combining these data sources should prevent the base model from forgetting how to represent multiple languages while it adapts to the target domain. experiments in the domains of financial text and biomedical text, across seven languages: french, german, spanish, romanian, portuguese, danish, and english, and on two downstream tasks: named entity recognition, and sentence classification, show the effectiveness of multilingual domain adaptive pretraining. further analysis in a cross-lingual biomedical sentence retrieval task indicates that mdapt enables models to learn better domain-specific representations, and that these representations transfer across languages. finally, we show that the difference in tokenizer quality between mono-and multilingual models is more pronounced in domain-specific text, indicating a direction for future improvement. all models trained with mdapt and the new datasets used in downstream tasks and pretraining data foot_0 and our code is made available foot_1 . ## problem formulation pretrained language models are trained from random initialization on a large corpus c of unlabelled sentences. each sentence is used to optimize the parameters of the model using a pretraining objective, for example, masked language modelling, where, for a given sentence, 15% of the tokens are masked in the input m, and the model is trained to predict those tokens @xcite . c is usually a corpus of no specific domain, 3 e.g. wikipedia or crawled web text. domain-adaptive pretraining is the process of continuing to pretrain a language model to suit a specific domain @xcite . this process also uses the masked language modelling pretraining objective, but the model is trained using a domain-specific corpus s, e.g. biomedical text if the model should be suited to the biomedical domain. our goal is to pretrain a single model, which will be used for downstream tasks in multiple languages within a specific domain, as opposed to having a separate model for each language. this single multilingual domain-specific model should, ideally, perform as well as language-specific domain-specific models in a domain-specific downstream task. in pursuit of this goal, we use different types of corpora for domain adaptive pretraining of a single multilingual model. each considered corpus has two properties: (1) a domain propertyit is a general or specific corpus; and (2) a language property -it is either monolinugal or multilingual. these properties can be combined, for example the multilingual wikipedia is a multi-general corpus, while the abstracts of english biomedical publications would be a monospecific corpus. recall that specific cor-3 text varies along different dimensions, e.g. topic or genre @xcite . in the context of this paper, we focus on domain-specificity along the topic dimension , i.e. texts are considered as domain-specific if they talk about a narrow set of related concepts. the domain-specific text can comprise different genres of text (e.g. financial news articles and financial tweets would both be considered as being from the financial domain). ## multilingual domain adaptive pretraining recall that we assume the availability of large scale english domain-specific and multilingual general unlabelled text. in addition to these monospecific and multi-general corpora, we collect multilingual domain-specific corpora, using two specific domains-financial and biomedicalas an example (section 3.1). note that although we aim to collect domain-specific data in as many languages as possible, the collected data are usually still relatively small. we thus explore different strategies to combine different data sources (section 3.2), resulting in three different types of pretraining corpora of around 10 million sentences, that exhibit specific and multi properties to different extents: e d : english domain-specific data; m d +e d : multilingual domain-specific data, augmented with english domain-specific data; and m d +m wiki : multilingual domain-specific data, augmented with multilingual general data. we use mbert @xcite as the multilingual base model, and employ two different continued pretraining methods (section 3.3): adapterbased training and full model training, on these three pretraining corpora, respectively. ## domain-specific downstream tasks to demonstrate the effectiveness of our multilingual domain-specific models, we conduct experiments on two downstream tasks-named entity recognition (ner) and sentence classificationusing datasets from biomedical and financial domains, respectively. ## results to measure the effectiveness of multilingual domain adaptive pretraining, we compare the effectiveness of our models trained with mdapt on downstream ner and classification, to the respective monolingual baselines (mono-general), and to the base multilingual model without mdapt ( baseline models as mono-general baselines, we use english bert @xcite , portuguese bert @xcite , romanian bert @xcite , beto @xcite for spanish, flaubert @xcite for french, german bert @xcite , and danish bert. 8 mono-specific baselines exist only for a few languages and domains, we use en-bio-bert @xcite as english biomedical baseline, and en-fin-bert @xcite as english financial baseline. to the best of our knowledge, pt-bio-bert @xcite is the only biomedical model for non-english language, we use it as portuguese biomedical baseline, see appendix a for more details. ## analysis our experiments suggest that mdapt results in a pretrained model which is better suited to solve domain-specific downstream tasks than mbert, and that mdapt narrows the gap to monolingual model performance. in this section, we present further analysis of these findings, in particular we investigate the quality of domain-specific representations learned by mdapt models compared to mbert, and the gap between mono-and multilingual model performance. ## related work recent studies on domain-specific bert @xcite @xcite , which mainly focus on english text, have demonstrated that in-domain pretraining data can improve the effectiveness of pretrained models on downstream tasks. these works continue pretraining the whole base model-bert or robertaon domain-specific corpora, and the resulting models are supposed to capture both generic and domain-specific knowledge. by contrast, @xcite ; @xcite ; @xcite train domain-specific models from scratch, tying an in-domain vocabulary. despite its effectiveness, this approach requires much more compute than domain adaptive pretraining, which our work focuses on. additionally, we explore an efficient variant of domain adaptive pretraining based on adapters @xcite , and observe similar patterns regarding pretraining a multilingual domain-specific model. several efforts have trained large scale multilingual language representation models using parallel data @xcite or without any cross-lingual supervision @xcite @xcite . however, poor performance on lowresource languages is often observed, and efforts are made to mitigate this problem @xcite @xcite . in contrast, we focus on the scenario that the nlp model needs to process domain-specific text supporting a modest number of languages. alternative approaches aim at adapting a model to a specific target task within the domain directly, e.g. by an intermediate supervised fine-tuning step @xcite , resulting in a model specialized for a single task. domain adaptive pretraining, on the other hand, aims at providing a good base model for different tasks within the specific domain. ## conclusion we extend domain adaptive pretraining to a multilingual scenario that aims to train a single multilingual model better suited for the specific domain. evaluation results on datasets from biomedical and financial domains show that although multilingual models usually underperform their monolingual counterparts, domain adaptive pretraining can effectively narrow this gap. on seven out of nine datasets for document classification and ner, the model resulting from multilingual domain adaptive pretraining outperforms the baseline multigeneral model, and on four it even outperforms the mono-general model. the encouraging results show the implication of deploying a single model which can process financial or biomedical documents in different languages, rather than building separate models for each individual language.