Forging Multiple Training Objectives for Pre-trained Language Models via Meta-Learning (2022)

Abstract: Multiple pre-training objectives fill the vacancy of the understanding capability of single-objective language modeling, which serves the ultimate purpose of pre-trained language models (PrLMs), generalizing well on a mass of scenarios. However, learning multiple training objectives in a single model is challenging due to the unknown relative significance as well as the potential contrariety between them. Empirical studies have shown that the current objective sampling in an ad-hoc manual setting makes the learned language representation barely converge to the desired optimum. Thus, we propose MOMETAS , a novel adaptive sampler based on meta-learning, which learns the latent sampling pattern on arbitrary pre-training objectives. Such a design is lightweight with negligible additional training overhead. To validate our approach, we adopt five objectives and conduct continual pre-training with BERT-base and BERT-large models, where MOMETAS demonstrates universal performance gain over other rule-based sampling strategies on 14 natural language processing tasks.

## introduction it is appealing for deep neural language models to generalize well on multiple downstream tasks through large-scale language pre-training, e.g. bert @xcite , electra @xcite , deberta @xcite and @xcite . most pre-trained language models (prlms) rely on only one or two pre-training objectives, from masked language modeling (mlm), next sentence prediction (nsp) @xcite , sentence order prediction (sop) @xcite and permutation language modeling (plm) @xcite . even though prlms are intended for high generalization, studies show that they are not always all-rounded and tend to be particularly weak in some aspects @xcite @xcite , while an ultimate purpose of a language understanding system is to stand for the nice initialization on a mass of scenarios simultaneously and effectively. with the birth of more and more pre-training objectives, a number of specific ones beyond are found of great benefit to enhance task-level understanding capability, e.g. contrastive learning @xcite , adversarial training @xcite , knowledge injection @xcite . to enjoy the merits of all worlds and let the model generalize better on more seen or perhaps unseen tasks, there naturally comes a need to combine all these objectives in an organic manner. however, learning multiple pre-training objectives simultaneously in a single model is challenging @xcite . a well-known issue is negative transfer @xcite in which learning well on one objective impairs another. more importantly, the relative significance between all objectives is supposed to be scheduled. for instance, nsp can take little effect on the model due to its simpleness in the mature stage of training. however, it is of great difficulty to heuristically tune such a ratio considering the large amounts of compute to pre-train once. in most cases we tentatively treat all of them equally @xcite , which makes the learned language representation barely converge to the optimal point and limits the model performance @xcite . to forge multiple training objectives for prlms, this paper presents to learn an optimal sampling strategy so that the more informative objective is more likely to be chosen. the backbone is metalearning @xcite and thus we call it multi-objective meta-sampler (mometas). in the proposed framework, we redesign the pretraining process into two phases, meta-train and meta-test. the model is trained alternately on one sampled objective at each step during metatrain, while the sampling distribution is then updated during meta-test by measuring the relative contribution of each objective. the training design is lightweight with little additional overhead to guarantee the pre-training efficiency. to validate our approach, we consider five pre-training objectives (e.g. for sentence embedding, knowledge caption, syntactic understanding) and continue to pre-train with bert-base and bert-large, where mometas demonstrates universal performance gain over other rule-based sampling strategies on 14 natural language processing tasks. ## multi-objective meta-sampler in this section, we first take an overview of our meta-learning framework. what follows is the preliminaries of the pre-training setting as well as a number of ruled-based samplers. then we discuss the details of our meta-sampler. ## experimental setup in this section, we present our experimental setup. our implementations are based on pytorch using transformers @xcite . ## empirical results glue more tasks we make further experiments on more different tasks as in ## ablation studies this section reports our ablation studies over a number of factors of mometas in order to better understand their roles. for all experiments, we report the results over five runs. ## conclusion this paper concentrates on multi-objective pretraining of prlms and presents multi-objective meta-sampler (mometas) in the hope of combining arbitrary pre-training objectives organically. we adopt five pre-training objectives and conduct experiments on the base-size and large-size models. the empirical results on a wide range of nlp tasks demonstrate that mometas largely outperforms other rule-based sampling strategies and unlocks more powerful language models.