  Current NLP models are predominantly trained through a pretrain-then-finetune
pipeline, where models are first pretrained on a large text corpus with a
masked-language-modelling (MLM) objective, then finetuned on the downstream
task. Prior work has shown that inserting an intermediate pre-training phase,
with heuristic MLM objectives that resemble downstream tasks, can significantly
improve final performance. However, it is still unclear (1) in what cases such
intermediate pre-training is helpful, (2) whether hand-crafted heuristic
objectives are optimal for a given task, and (3) whether a MLM policy designed
for one task is generalizable beyond that task. In this paper, we perform a
large-scale empirical study to investigate the effect of various MLM policies
in intermediate pre-training. Crucially, we introduce methods to automate
discovery of optimal MLM policies, by learning a masking model through either
direct supervision or meta-learning on the downstream task. We investigate the
effects of using heuristic, directly supervised, and meta-learned MLM policies
for intermediate pretraining, on eight selected tasks across three categories
(closed-book QA, knowledge-intensive language tasks, and abstractive
summarization). Most notably, we show that learned masking policies outperform
the heuristic of masking named entities on TriviaQA, and masking policies
learned on one task can positively transfer to other tasks in certain cases.
