Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020)

Abstract: Language models pretrained on text from a wide variety of sources form the foundation of today’s NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task’s unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multi-phase adaptive pretraining offers large gains in task performance.

## introduction today's pretrained language models are trained on massive, heterogeneous corpora @xcite . for instance, roberta @xcite was trained on over 160gb of uncompressed text, with sources ranging from englishlanguage encyclopedic and news articles, to literary works and web content. representations learned by such models achieve strong performance across many tasks with datasets of varying sizes drawn from a variety of sources (e.g., @xcite . this leads us to ask whether a task's textual domain-a term typically used to denote a distribution over language characterizing a given topic or genre (such as "science" or "mystery novels")-is still relevant. do the latest large pretrained models work universally or is it still helpful to build figure 1 : an illustration of data distributions. task data is comprised of an observable task distribution, usually non-randomly sampled from a wider distribution (light grey ellipsis) within an even larger target domain, which is not necessarily one of the domains included in the original lm pretraining domain -though overlap is possible. we explore the benefits of continued pretraining on data from the task distribution and the domain distribution. while some studies have shown the benefit of continued pretraining on domain-specific unlabeled data (e.g., @xcite , these studies only consider a single domain at a time and use a language model that is pretrained on a smaller and less diverse corpus than the most recent language models. moreover, it is not known how the benefit of continued pretraining may vary with factors like the amount of available labeled task data, or the proximity of the target domain to the original pretraining corpus (see figure 1 ). we address this question for one such highperforming model, roberta @xcite ( §2). we consider four domains (biomedical and computer science publications, news, and reviews; §3) and eight classification tasks (two in each domain). for targets that are not already in-domain for roberta, our experiments show that contin-ued pretraining on the domain (which we refer to as domain-adaptive pretraining or dapt) consistently improves performance on tasks from the target domain, in both high-and low-resource settings. above, we consider domains defined around genres and forums, but it is also possible to induce a domain from a given corpus used for a task, such as the one used in supervised training of a model. this raises the question of whether pretraining on a corpus more directly tied to the task can further improve performance. we study how domainadaptive pretraining compares to task-adaptive pretraining, or tapt, on a smaller but directly taskrelevant corpus: the unlabeled task dataset ( §4), drawn from the task distribution. task-adaptive pretraining has been shown effective @xcite , but is not typically used with the most recent models. we find that tapt provides a large performance boost for roberta, with or without domain-adaptive pretraining. finally, we show that the benefits from taskadaptive pretraining increase when we have additional unlabeled data from the task distribution that has been manually curated by task designers or annotators. inspired by this success, we propose ways to automatically select additional task-relevant unlabeled text, and show how this improves performance in certain low-resource cases ( §5). on all tasks, our results using adaptive pretraining techniques are competitive with the state of the art. in summary, our contributions include: one such pretrained lm is roberta @xcite , which uses the same transformerbased architecture @xcite as its predecessor, bert @xcite . it is trained with a masked language modeling objective (i.e., cross-entropy loss on predicting randomly masked tokens). the unlabeled pretraining corpus for roberta contains over 160 gb of uncompressed raw text from different english-language corpora (see appendix §a.1). roberta attains better performance on an assortment of tasks than its predecessors, making it our baseline of choice. although roberta's pretraining corpus is derived from multiple sources, it has not yet been established if these sources are diverse enough to generalize to most of the variation in the english language. in other words, we would like to understand what is out of roberta's domain. towards this end, we explore further adaptation by continued pretraining of this large lm into two categories of unlabeled data: (i) large corpora of domain-specific text ( §3), and (ii) available unlabeled data associated with a given task ( §4). ## domain-adaptive pretraining our approach to domain-adaptive pretraining (dapt) is straightforward-we continue pretraining roberta on a large corpus of unlabeled domain-specific text. the four domains we focus on are biomedical (biomed) papers, computer science (cs) papers, newstext from realnews, and amazon reviews. we choose these domains because they have been popular in previous work, and datasets for text classification are available in each. ## task-adaptive pretraining datasets curated to capture specific tasks of interest tend to cover only a subset of the text available within the broader domain. for example, the chemprot dataset for extracting relations between chemicals and proteins focuses on abstracts of recently-published, high-impact articles from hand-selected pubmed categories @xcite . we hypothesize that such cases where the task data is a narrowly-defined subset of the broader domain, pretraining on the task dataset itself or data relevant to the task may be helpful. task-adaptive pretraining (tapt) refers to pretraining on the unlabeled training set for a given task; prior work has shown its effectiveness (e.g. @xcite . compared to domainadaptive pretraining (dapt; §3), the task-adaptive approach strikes a different trade-off: it uses a far smaller pretraining corpus, but one that is much more task-relevant (under the assumption that the training set represents aspects of the task well). this makes tapt much less expensive to run than dapt, and as we show in our experiments, the performance of tapt is often competitive with that of dapt. ## augmenting training data for task-adaptive pretraining in §4, we continued pretraining the lm for task adaptation using only the training data for a supervised task. inspired by the success of typically curated by humans. we explore two scenarios. first, for three tasks (rct, hyperpartisan, and imdb) we use this larger pool of unlabeled data from an available human-curated corpus ( §5.1). next, we explore retrieving related unlabeled data for tapt, from a large unlabeled in-domain corpus, for tasks where extra human-curated data is unavailable ( §5.2). ## conclusion we investigate several variations for adapting pretrained lms to domains and tasks within those domains, summarized in