Pre-train or Annotate? Domain Adaptation with a Constrained Budget (2021)

Abstract: Recent work has demonstrated that pre-training in-domain language models can boost performance when adapting to a new domain. However, the costs associated with pre-training raise an important question: given a fixed budget, what steps should an NLP practitioner take to maximize performance? In this paper, we study domain adaptation under budget constraints, and approach it as a customer choice problem between data annotation and pre-training. Specifically, we measure the annotation cost of three procedural text datasets and the pre-training cost of three in-domain language models. Then we evaluate the utility of different combinations of pre-training and data annotation under varying budget constraints to assess which combination strategy works best. We find that, for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, a combination of data annotation and in-domain pre-training works more optimally. We therefore suggest that task-specific data annotation should be part of an economical strategy when adapting an NLP model to a new domain.

## introduction the conventional wisdom on semi-supervised learning and unsupervised domain adaptation is that labeled data is expensive; therefore, training on a combination of labeled and unlabeled data is an economical approach to improve performance when adapting to a new domain (blum and mitchell, 1998; daume iii and marcu, 2006; hoffman et al., 2018; chen et al., 2020). recent work has shown that pre-training in-domain transformers is an effective method for unsupervised adaptation @xcite our code and data are publicly available on github: https://github.com/bflashcp3f/procbert . optimal choice number of pretraining hours (tpus) number of annotated documents (in-domain) budget constraint @xmath0.7 @xmath1.8 @xmath2.9 x a c a + x p c p  b figure 1: we view domain adaptation as a consumer choice problem (becker, 1965; lancaster, 1966). the nlp practitioner (consumer) is faced with the problem of choosing an optimal combination of annotation and pre-training under a constrained budget. this figure is purely for illustration and is not based on experimental data. augenstein, 2020) and even boosts performance when large quantities of in-domain data are available (gururangan et al., 2020). however, modern pre-training methods incur substantial costs (izsak et al., 2021), and generate carbon emissions (strubell et al., 2019; schwartz et al., 2020; bender et al., 2021). this raises an important question: given a fixed budget to improve a model's performance, what steps should an nlp practitioner take? on one hand, they could hire annotators to label in-domain task-specific data, while on the other, they could buy or rent gpus or tpus to pre-train large in-domain language models. in this paper, we empirically study the best strategy for adapting to a new domain given a fixed budget. we view the nlp practitioner's dilemma of how to adapt to a new domain as a problem of consumer choice, a classical problem in microeconomics (becker, 1965; lancaster, 1966). as illustrated in figure 1 , the nlp practitioner (consumer) can obtain x a annotated documents (by hiring annotators) at a cost of c a each, and x p hours of pre-training (by renting gpus or tpus) at a cost of c p per hour. given a fixed budget b, the consumer may choose any combination that fits within the budget constraint x a c a + x p c @xmath3. the goal is to choose a combination that maximizes the utility function, u (x a , x p ), which can be defined using an appropriate performance metric, such as f 1 score, that is achieved after pre-training for x p hours and then fine-tuning on x a in-domain documents. to empirically estimate the cost of annotation, we hire annotators to label domain-specific documents for supervised fine-tuning in three procedural text domains: wet-lab protocols, paragraphs describing scientific procedures in pubmed articles, and chemical synthesis procedures described in patents. we choose to target natural language understanding for scientific procedures in this study, because there is an opportunity to help automate lab protocols and support more reproducible scientific experiments, yet few annotated datasets currently exist in these domains. furthermore, annotation of scientific procedures is not easily amenable to crowdsourcing, making this an ideal testbed for pretraining-based domain adaptation. we measure the cost of in-domain pre-training on a large collection of unlabeled procedural texts using google's cloud tpus. @xcite model performance is then evaluated under varying budget constraints in six source and target domain combinations. our analysis suggests that given current costs of pre-training large transformer models, such as bert (devlin et al., 2019), and roberta (liu et al., 2019), in-domain data annotation should always be part of an economical strategy when adapting a single nlp system to a new domain. for small budgets (e.g. less than $800 usd), spending all funds on annotation is the best policy; however, as more funding becomes available, a combination of pre-training and annotation is the best choice. this paper addresses a specific question that is often faced by nlp practitioners working on applications: what is the most economical approach to adapt an nlp system to a new domain when no pre-trained models or task-annotated datasets are initially available? if multiple nlp systems need to be adapted to a single target domain, model costs can be amortized, making pre-training an attractive option for smaller budgets. ## scope of the study in this study, we focus on a typical scenario faced by an nlp practitioner to adapt a single nlp sys-tem to a single new domain, maximizing performance within a constrained budget. we consider only the direct benefit on the target task in our main analysis ( §5), however we do provide additional analysis of positive externalities on other related tasks that may benefit from a new pre-trained model in §6. we estimate cost based on two major expenses: annotating task-specific data ( §3) and pre-training domain-specific models using tpus ( §4). note that fine-tuning costs are not included in our analysis, as they are nearly equal whether budget is invested into pre-training or annotation. @xcite we assume a generic bert is the closest zero-cost model that is initially available, which is likely the case in real-world domain adaptation scenarios (especially for non-english languages). our experiments are designed to simulate a scenario where no domainspecific model is initially available. we also assume that the nlp engineer's salary is a fixed cost; in other words, their salary will be the same whether they spend time pre-training models or managing a group of annotators. @xcite our primary concerns are about financial and environmental costs, rather than the overall time needed to obtain the adapted model. if the timeline is an important factor, the annotation process can be possibly sped up by hiring more annotators. ## estimating annotation cost (c a ) in this section, we present our estimates of the annotation cost for three procedural text datasets from specialized scientific domains, which enable a comparison of model performance under varying budget constraints ( §5.4). annotated procedural text datasets. we experiment with three procedural text corpora, including wet lab protocols (wlp; tabassum et al., 2020) and two new datasets we created for this study, which include scientific articles and chemical patents. statistics of the three datasets are shown in ## positive externalities of pretraining so far, we have discussed domain adaptation as a consumer choice problem where annotation and pre-training costs are balanced to maximize performance in a target domain. however, pre-training on large quantities of natural language instructions can improve performance on additional tasks in the procedural text domain, as demonstrated in the following subsections. ## conclusion in this paper, we address a number of questions related to the costs of adapting an nlp model to a new domain (blitzer et al., 2006; han and eisenstein, 2019), an important and well-studied problem in nlp. we frame domain adaptation under a constrained budget as a problem of consumer choice. experiments are conducted using several pre-trained models in three procedural text domains to determine when it is economical to pre-train indomain transformers (gururangan et al., 2020), and when it is better to spend available resources on annotation. our results suggest that when a small number of nlp models need to be adapted to a new domain, pre-training, by itself, is not an economical solution.