Pre-training Language Model as a Multi-perspective Course Learner (2023)

Abstract: ELECTRA, the generator-discriminator pre-training framework, has achieved impressive semantic construction capability among various downstream tasks. Despite the convincing performance, ELECTRA still faces the challenges of monotonous training and deficient interaction. Generator with only masked language modeling (MLM) leads to biased learning and label imbalance for discriminator, decreasing learning efficiency; no explicit feedback loop from discriminator to generator results in the chasm between these two components, underutilizing the course learning. In this study, a multi-perspective course learning (MCL) method is proposed to fetch a many degrees and visual angles for sample-efficient pre-training, and to fully leverage the relationship between generator and discriminator. Concretely, three self-supervision courses are designed to alleviate inherent flaws of MLM and balance the label in a multi-perspective way. Besides, two self-correction courses are proposed to bridge the chasm between the two encoders by creating a “correction notebook” for secondary-supervision. Moreover, a course soups trial is conducted to solve the “tug-of-war” dynamics problem of MCL, evolving a stronger pre-trained model. Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style models under the same settings. The pre-trained MCL model is available at https://huggingface.co/McmanusChen/MCL-base .

## introduction language models pre-training @xcite @xcite has shown great success in endowing machines with the ability to understand and process various downstream nlp tasks. a wide range of pre-training strategies have been proposed, among which the most prevailing object is the masked language modeling (mlm) @xcite . such autoencoding language modeling objects @xcite typically first randomly corrupt a certain percentage of training corpus with masked tokens, and then encourage encoders to restore the original corpus. to reduce the randomness of pre-training and produce a sample-efficient method, electrastyle frameworks @xcite ) leverage an transformer-based @xcite generator training with mlm to build challenging ennoising sentences for a discriminator in the similar structure to carry out the denoising procedure. typically in the electra-style training, the generator first constructs its semantic representations through mlm training and cloze these masked sentences with pseudo words; in the meanwhile, the discriminator inherits the information from the former and distinguish the originality of every token, which is like a step-by-step course learning. however, only mlm-based generator training may lead to monotonous learning of data, which conduces to the incomprehensive generation and imbalanced label of corrupted sentences for the discriminator @xcite . besides, interactivity between the two encoders stops abruptly except the sharing of embedding layers @xcite , since there is no direct feedback loop from discriminator to generator. to enhance the efficiency of training data and to adequately utilize the relationship of generator and discriminator, in this work, we propose a sampleefficient method named multi-perspective course learning (mcl). in the first phase of mcl, to fetch a many degrees and visual angles to impel initial semantic construction, three self-supervision courses are designed, including cloze test, word rearrangement and slot detection. these courses instruct language models to deconstruct and dissect the exact same corpus from multi perspectives under the electra-style framework. in the second phase, two self-correction courses are tasked to refine both generator and discriminator. a confusion matrix regarding to discriminator's recognition of each sentence is analyzed and applied to the construction of revision corpora. secondary learning is carried out for the two components in response to the deficiencies in the previous course learning. at last, the model mines the same batch of data from multiple perspectives, and implement progressive semantic learning through the self-correction courses. experiments on the most widely accepted benchmarks glue @xcite and squad 2.0 @xcite demonstrate the effectiveness of the proposed mcl. compared with previous advanced systems, mcl achieved a robust advantage across various downstream tasks. abundant ablation studies confirm that multi-perspective courses encourage models to learn the data in a sample-efficient way. besides, a course soups trial is conducted to further interpret and dissect the core of multi-perspective learning, providing a novel approach to enhance the pre-training efficiency and performance. ## preliminary in this work, we built our system based on the electra-style framework. thus, the framework of electra is reviewed. unlike bert @xcite , which uses only one transformer encoder trained with mlm, electra is trained with two transformer encoders: a generator g and a discriminator d. g is trained with mlm and used to generate ambiguous tokens to replace masked tokens in the input sequence. then the modified input sequence is fed to d, which needs to determine if a corresponding token is either an original token or a token replaced by the generator. generator training formally, given an input sequence @xmath0, x 2 , ..., x n ], a mask operation is conducted to randomly replace its tokens with [mask] at the position set r. foot_0 and the masked sentence x @xmath1, x 2 , ..., [mask] i , ..., x n ] is fed into the generator to produce the contextualized representations {h i } n @xmath2. g is trained via the following loss l mlm to predict the original tokens from the vocabulary v at the masked positions: ## challenges biased learning though the electra training method is simple and effective, treating corpora from a single perspective could cause biased learning. as for the progress of mlm and rtd, there exists an inherent flaw that g might predict appropriate but not original token on the [mask] position, and such appropriate expression still needs to be judged as substitution by d. for example, if the original sequence "alan buys an apple" is masked as "alan [mask] an apple", there are too many candidate words like "eats, peels, cuts" that can replace @xcite to form a harmonious context. thus, to request d to continue to distinguish replaced tokens is a difficult even awkward task. treating the same piece of data in a single way reduces the effectiveness of training. as for the distribution of generated corpus from g, the labelimbalance may gradually emerge with the mlm training of g, which could disturb the rtd training of d. as the semantic construction of g thrives with the pre-training, the pseudo token on [mask] becomes more reasonable and even matches the original word. thus, the proportion of replaced tokens in the training sentences of d collapses, which interferes with the binary classification task seriously. in this work, three self-supervision courses are tasked to train the model in a multi-perspective way, improving the learning efficiency of data and balancing the distribution of labels. ## methodology in this section, we will start by formulating three self-supervision courses which encourage models to treat data in a multi-perspective way. then two self-correction courses are elaborated, deriving from the course-like relationship between g and d. these various courses are weaved into the entirety of the multi-perspective course learning method. ## conclusion this paper proposes the multi-perspective course learning method, containing three self-supervision courses to improve learning efficiency and balance label distributions, as well as two self-correction courses to create a "correction notebook" for revision training. besides, the course soups method is designed to figure out a novel approach for efficient pre-training. experiments show that our method significantly improves electra's performance and overshadows multiple advanced models under same settings, verifying the effectiveness of mcl.