On the Influence of Masking Policies in Intermediate Pre-training (2021)

Abstract: Current NLP models are predominantly trained through a two-stage “pre-train then fine-tune” pipeline. Prior work has shown that inserting an intermediate pre-training stage, using heuristic masking policies for masked language modeling (MLM), can significantly improve final performance. However, it is still unclear (1) in what cases such intermediate pre-training is helpful, (2) whether hand-crafted heuristic objectives are optimal for a given task, and (3) whether a masking policy designed for one task is generalizable beyond that task. In this paper, we perform a large-scale empirical study to investigate the effect of various masking policies in intermediate pre-training with nine selected tasks across three categories. Crucially, we introduce methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. We conclude that the success of intermediate pre-training is dependent on appropriate pre-train corpus, selection of output format (i.e., masked spans or full sentence), and clear understanding of the role that MLM plays for the downstream task. In addition, we find our learned masking policies outperform the heuristic of masking named entities on TriviaQA, and policies learned from one task can positively transfer to other tasks in certain cases, inviting future research in this direction.

## introduction large, neural language models (lms) pre-trained with masked language modeling @xcite have achieved impressive results over a variety of nlp tasks. studies show that an additional intermediate pre-training stage between general pre-training and task-specific finetuning further improves downstream performance (fig. 1 ). for example, intermediate pre-training † work partially done while interning at facebook ai. ## preliminary: masked language modeling in this section, we revisit mlm objective with the notation that we will use throughout the paper. mlm is a predominant pre-training objective for large-scale transformers in nlp @xcite . mlm and its variants can be characterized with two key components: a masking policy g(.; φ), parameterized by φ, which decides the collection of tokens to be masked, and a language model f (.; θ), parameterized by θ. formally, given a sequence of tokens @xmath0, x 2 , ..., x m ], g(x; φ) generates a sequence of binary decisions @xmath1, d 2 , ..., d m ], where d @xmath2. the source sequence for pre-training, x (src) , is formulated by replacing the selected tokens with a @xmath3, i.e., we denote this operation as x (src) = x ⊕ d. the target sequence x (tar) can be either the full original sequence x (bart, @xcite , or the sequence of masked tokens (t5, @xcite . ## analysis setup in this section we introduce the analysis pipeline ( §3.1) and downstream datasets we use ( §3.2). we defer the details of learned masking policies to §4. ## compared masking policies we experiment with three categories of masking policies: heuristic policies, where g is a fixed heuristic function ( §4.1); supervised policies, where g is a model whose weights are learned from direct supervision on downstream tasks ( §4.2); and meta-learned policies, where g is a model whose weights are learned through meta-learning on downstream tasks ( §4.3). ## results and discussion following our analysis setup ( §3), we present the results for closed-book qa in ## conclusion in this paper, we study the influence brought by different masking policies used during intermediate pre-training, and offer two methods as our initial attempts towards automating the discovery of optimal masking policy. from extensive experiments with heuristic and learned masking policies across three categories of tasks, we have identified several successful cases of intermediate pre-training, offered in-depth analysis and insights for the masking policies we used, discussed the risks of learned masking policies, and summarized several suggestions for researchers who wish to adopt intermediate pre-training in their applications. we also acknowledge that, despite our additional efforts and experiments, several observations still cannot be explained away. we invite future research into this challenging and under-explored problem, to expand on our methods, and to search the space of pre-training objectives beyond masked language modeling. furthermore, we hope our work encourages researchers to consider the type of downstream applications they wish to deploy their lms in, before investing resources into largescale pre-training.