Multidomain Pretrained Language Models for Green NLP (2021)

Abstract: When tackling a task in a given domain, it has been shown that adapting a model to the domain using raw text data before training on the supervised task improves performance versus solely training on the task. The downside is that a lot of domain data is required and if we want to tackle tasks in n domains, we require n models each adapted on domain data before task learning. Storing and using these models separately can be prohibitive for low-end devices. In this paper we show that domain adaptation can be generalised to cover multiple domains. Specifically, a single model can be trained across various domains at the same time with minimal drop in performance, even when we use less data and resources. Thus, instead of training multiple models, we can train a single multidomain model saving on computational resources and training time.

## introduction domain adaptation in the form of training on unlabeled data has been prevalent in recent work @xcite . when given a task t in domain d, it is useful to adapt our model on raw text data pertinent to d before supervised training on the labeled data of t . unfortunately, domain adaptation is expensive and costly. so even though results do get better, there needs to be a balance between use of computational resources and model performance. accentuating the issue is the rising dominance of increasingly larger models @xcite . these large models require a lot of data and computational resources to train. not only have these resources been prohibitive for smaller labs, but the environmental impact of training such large models cannot be understated either @xcite . there is therefore a need to build models that can tackle tasks across multiple domains, much in the same way multilingual models are able to operate across multiple languages. these multidomain models, to be useful, need to exhibit performance comparable to models adapted to a single domain. then, these models can be trained and deployed with reduced computational costs. in this paper we explore such multidomain models. we compare multidomain distilbert @xcite models with single-domain distil-bert models 1 . in our analysis, we test the multidomain model on tasks from multiple domains (including multinli), we examine how important adaptation order is for performance and we show that training on the domains jointly or sequentially does not impact effectiveness. we also reproduce findings in @xcite , where the authors showed that pretraining on an irrelevant domain is not beneficial. finally, we show that much like multilingual bert, multidomain models can be used for better performance in low-resource domains, with the low-resource domain leveraging the larger pretrained model. ## related work it has been shown empirically that domain adaptation improves model performance @xcite . specifically, the authors experimented on roberta @xcite and eight tasks ranging across four domains (computer science and biomedical papers, reviews and news). performance increased when adapting to a domain pertinent to the domain of the task, compared to when the two domains were not as related. work has also been done to alleviate the environmental impact of training larger models. in @xcite , an inexpensive method was proposed for domain adaptation, which can be performed on a cpu and significantly reduces training cost. finally, mobilebert is a downsized bert model with around 4 times fewer parameters @xcite than the original bert-base model. there is a need to fit larger models in low-capacity machines, with mobilebert and our work being a step in that direction, alongside other work in the pruning area @xcite . ## training the training of our models is broken up in two steps: domain adaptation and supervised task learning. furthermore, we have two setups for our ex-periments: a) perform domain adaptation on four models in parallel and then train them on each individual task, and b) perform domain adaptation on a single model for all domains successively before task learning. we also perform ablation studies on irrelevant and low-resource domain adaptation, domain adaptation order and finally investigate whether joint instead of sequential adaptation performs better. ## conclusion in this work we show that domain adaptation can be extended to multiple domains. these multidomain models are able to tackle tasks across various domains with minimal performance drop compared to single-domain models, while using fewer resources and reducing our carbon footprint. in addition, based on @xcite , we can use several finetuned instances of a multidomain model for a number of tasks, with negligible increase in memory usage. so our multidomain models are also beneficial on low-resource devices. finally, we show that adapting to multiple domains always provides a performance increase and that tasks in low-resource domains receive a boost from multidomain models.