Instance Regularization for Discriminative Language Model Pre-training (2022)

Abstract: Discriminative pre-trained language models (PrLMs) can be generalized as denoising auto-encoders that work with two procedures, ennoising and denoising. First, an ennoising process corrupts texts with arbitrary noising functions to construct training instances. Then, a denoising language model is trained to restore the corrupted tokens. Existing studies have made progress by optimizing independent strategies of either ennoising or denosing. They treat training instances equally throughout the training process, with little attention on the individual contribution of those instances. To model explicit signals of instance contribution, this work proposes to estimate the complexity of restoring the original sentences from corrupted ones in language model pre-training. The estimations involve the corruption degree in the ennoising data construction process and the prediction confidence in the denoising counterpart. Experimental results on natural language understanding and reading comprehension benchmarks show that our approach improves pre-training efficiency, effectiveness, and robustness. Code is publicly available at https://github.com/cooelf/InstanceReg.

## introduction leveraging self-supervised objectives to pre-train language models (prlms) on massive unlabeled data has shown success in natural language processing (nlp) @xcite @xcite @xcite @xcite . a wide landscape of pre-training objectives has been produced, such as autoregressive @xcite and autoencoding @xcite * work done during internship at lanboat. this work was partially supported by key projects of national natural science foundation of china (u1836222 and 61733011). ennoising denoising [mask] cute dog [mask] playing on the [mask] ... a cute [mask] is [mask] on the [mask] ... original sentence corrupted sentence restored sentence discriminative training joshi et al., 2020) language modeling objectives, which serve as the principled mechanisms to teach language models general-purpose knowledge through the pre-training, and then those pretrained prlms can be fine-tuned for downstream tasks. based on these unsupervised functions, three classes of prlms have been proposed: autoregressive language models (e.g. gpt @xcite ), autoencoding models (e.g. bert @xcite ), and encoderdecoder models (e.g. bart @xcite and t5 @xcite ). in this work, we focus on the research line of autoencoding models, also known as discriminative prlms that have achieved impressive performance on natural language understanding (nlu). although the discriminative prlms may vary in language modeling functions or architectures as discussed above, they can be generalized as denoising auto-encoders, which contain two procedures, ennoising and denoising. the pretraining procedure is illustrated in figure 1 . most of the existing studies of prlms fall into the scope of either investigating better ennoising operations or more effective denoising strategies. they treat training instances equally throughout the training process. little attention is paid to the individual contribution of those instances. in standard mlm ennoising, randomly masking different tokens would lead to different degrees of corruption that may, therefore, cause different levels of difficulty in sentence restoration in denoising (as shown in figure 1 ) and thus increase the uncertainty in restoring the original sentence structure during the denoising process. for example, if "not" is masked, the corrupted sentence tends to have a contrary meaning. in this work, we are motivated to estimate the complexity of restoring the original sentences from corrupted ones in language model pretraining, to provide explicit regularization signals to encourage more effective and robust pre-training. our approach includes two sides of penalty: 1) ennoising corruption penalty that measures the distribution disparity between the corrupted sentence and the original sentence, to measure the corruption degree in the ennoising process; 2) denoising prediction penalty that measures the distribution difference between the restored sequence and the original sentence to measure the sentence-level prediction confidence in the denoising counterpart. experiments show that language models trained with our regularization terms can yield better performance and become more robust against adversarial attacks. ## related work training powerful large-scale language models on a large unlabeled corpus with self-supervised objectives has attracted lots of attention, which commonly work in two procedures of ennoising and denoising. the most representative task for pre-training is mlm, which is introduced in @xcite to pre-train a bidirectional bert. a spectrum of ennoising extensions has been proposed to enhance mlm further and alleviate the potential drawbacks, which fall into two categories: 1) mask units and 2) noising scheme. mask units correspond to the language modeling units that serve as knowledge carriers in different granularity. the variants focusing on mask units include the standard subword masking @xcite , span masking @xcite , and n-gram masking @xcite . for noising scheme, bart @xcite ) corrupts text with arbitrary noising functions, including token deletion, text infilling, sentence permutation, in conjunction with mlm. unilm @xcite extends the mask prediction to generation tasks by adding the auto-regressive objectives. xlnet @xcite proposes the permuted language modeling to learn the dependencies among the masked tokens. macbert @xcite suggests using similar words for the masking purpose. @xcite also investigates simple pre-training objectives based on token-level classification tasks as replacements of mlm, which are often computationally cheaper and result in comparable performance to mlm. in addition, electra @xcite proposes a novel training objective called replaced token detection, which is defined over all input tokens. although the above studies have an adequate investigation to reduce the mismatch between pre-training and fine-tuning tasks, an essential problem of the common denoising mechanism lacks attention. the construction of training examples based on ennoising operations would cause the break of sentence structure, either for replacement, addition, or deletion-based noising functions. in extreme cases, the destruction would lead to completely different sentences, making it difficult for the model to predict the corrupted tokens. therefore, in this work, we propose to enhance the pre-training quality by using instance regularization (ir) terms to estimate the restoration complexity from both sides of ennoising and denoising aspects. the proposed approach is partially related to some prior studies of hardness measurement in training deep learning models @xcite @xcite , whose focus is to guide the model to pay special attention to hard examples and prevent the vast number of easy negatives from overwhelming the training process. in contrast to optimizing the training process by heuristically finding the hard negatives, this work does not need to distinguish hard examples from ordinary ones, but measures the corruption degree between the masked sentence and the original sentence instead, and uses the degree as the explicit training signals. ## methodology this section will start by formulating the ennoising and denoising processes for building prlms and then introduce our instance regularization approach to estimate the restoration complexity in both ennoising and denoising views. ## analysis 5.1 ablation study to investigate the contribution of the internal components of the proposed ir objective, we conduct an ablation study under bert base and electra base on the glue development set. ## limitations we acknowledge that the major limitation of the proposed method is additional computation compared with the vanilla language models because the calculation of the regularization terms involves two forward passes of input sequences. as discussed in section 5.5, the training time of bert-ir base and bert base baseline for 200k steps is 67h/74h (about 10% increase) with the same hyper-parameters on base models. a more efficient instance regularization method without additional training passes could be future work.