PEER : Pre-training ELECTRA Extended by Ranking (2023)

Abstract: The BERT model and its variants have made great achievements in many downstream natural language processing tasks. The achievements of these models, however, demand highly expensive pre-training computation cost. To address this pre-training efficiency issue, the ELECTRA model is proposed to use a discriminator to perform replaced token detection (RTD) task, that is, to classify whether each input token is original or replaced by a generator. The RTD task performed by the ELECTRA accelerates pre-training so substantially, such that it is very challenging to further improve the pre-training efficiency established by the ELECTRA by using or adding other pre-training tasks, as the recent comprehensive study of Bajaj et al. (2022) summarizes. To further advance this pre-training efficiency frontier, in this paper we propose to extend the RTD task into a task of ranking input tokens according to K different quality levels. Essentially, we generalize the binary classifier in the ELECTRA into a K-level ranker to undertake a more precise task with negligible additional computation cost. Our extensive experiments show that our proposed method is able to outperform the state-of-the-art pre-training efficient models including ELECTRA in downstream GLUE tasks given the same computation cost.

## introduction language model pre-training has made great achievements in many natural language processing (nlp) downstream tasks, by designing and using effective pre-training tasks. a milestone is the bert model, which conducts a masked language model (mlm) task by randomly masking a proportion (typically 15%) of tokens in the input sentence and then recover the original sentence. since the success of the bert, many variant models @xcite @xcite @xcite have been proposed to further improve the performance of the bert by refining or adding pre-training tasks. one common issue of these mlm-based models, however, is the pre-training efficiency, because they all need highly expensive pre-training computation cost to achieve good performance. to address this issue, the electra model is proposed by @xcite . the electra uses an auxiliary generator network to provide plausible tokens to replace a proportion (typically 15%) of the original tokens according to the input context, and then utilizes a main discriminator network to perform replaced token detection (rtd) task, that is, to classify whether each token is original or replaced by the generator. in order to prevent the generator from producing replaced tokens overchallenging for the training of discriminator, the electra make the generator relative weaker than the discriminator by decreasing the hidden size of the generator. after its pre-training, the generator is discarded and the discriminator is further finetuned for downstream nlp tasks. the electra has shown impressive advantages over mlm-based models in various downstream tasks under similar computation cost, especially when a model size is small. after the success of the electra, researchers have proposed quite a few models each of which has an auxiliary generator network. because the rtd task performed by the electra has accelerated pre-training so substantially, however, it is very challenging to advance the efficiency frontier established by the electra, by using or adding other pre-training tasks, as the recent comprehensive study of @xcite summarizes. thus, to further improve the pre-training efficiency, we propose to extend the rtd task in the electra into a token quality ranking (tqr) task, a task of ranking input tokens according to k different quality levels. besides determining whether each input token is replaced by a generator or not, the tqr task also needs to distinguish replaced tokens by ranking them according to their replacement quality. we call our method peer, pre-training electra extended by ranking. please refer to figure 1 for demonstration. our proposal is based on the key observation that the quality of replaced tokens are not even. while some replaced tokens fit the context nearly as well as the corresponding original tokens, others do not. thus, our peer generalizes the binary classifier in the electra into a k-level ranker to perform a more precise task. we design a scheme capable of retrieving rank labels for a majority of replaced tokens from the relative weak generator, which serves as the basis for the tqr task. the extension from the electra to the peer also adds negligible computation cost, because the tqr task largely re-uses the computation already performed by the original electra. additionally, our peer adopts partial transformer-layer sharing technique between generator and ranker to further reduce computation cost in our method, as its advantage has been demonstrated in the teams model @xcite , a recent model proposed to improve the elec-tra. our extensive experiments in small and base scale models show that the peer is able to outperform both the electra and the teams in downstream glue tasks using the same or less computation cost. ## related work as introduced in section 1, since electra @xcite greatly boosts the pre-training efficiency, a few models have been proposed in order to further advance this pre-training efficiency frontier. the electric model is proposed by @xcite as an energy-based model to perform the cloze task @xcite using noise-contrastive estimation @xcite . it is particularly effective at producing likelihood scores for text but slightly under-performs electra on the glue tasks. the mc-bert model is proposed by @xcite to replace the rtd binary classification task in electra with a multi-choice cloze test with a reject option (which is essentially a multi-class classification task). the mc-bert consists of a meta controller network and a generator network. the meta controller corrupts the original input sentence by replacing a proportion of tokens with sam-pled tokens, just as electra's generator does. meanwhile, the meta controller also generates a set of k candidate tokens for each token in the input sentence. the generator uses the corrupted sentence as the input and learns to correct each token by choosing the correct answer among its k candidates. @xcite empirically show that the overall performance of the mc-bert is similar to that of the electra in glue tasks, since the mc-bert outperforms the electra in glue semantic tasks but is worse than the electra in the glue syntactic task cola. coco-lm @xcite is proposed to improve electra by using two new pre-training tasks called corrective language modeling (clm) task and sequence contrastive learning (scl) task. while electra's main network (discriminator) conducts only rtd task for each token position, coco-lm's main network undertakes the clm task by jointly performing both rtd task and mlm task for each token position in the corrupted input. additionally, coco-lm's main network also performs the scl task to find a pair of the mlm replaced sentence and the cropped sentence originated from the same source sentence among all other sentences in the same training batch. the debertav3 @xcite is proposed to combine both the advantages of the deberta model @xcite and those of the elec-tra. the deberta @xcite introduces two novel mechanisms to improve the effectiveness of the mlm task: disentangled attention and an enhanced mask decoder. the disentangled attention computes the attention weights among tokens using disentangled matrices on two separate vectors (content vector and relative position vector) of each token, while an enhanced mask decoder includes absolute positions in the decoding layer to predict the masked tokens. the debertav3 keeps these mechanisms but replaces the mlm task (used in the deberta) with electra's rtd task, and shows that the new combination outperforms both the original deberta and the electra. additionally, the debertav3 introduces gradientdisentangled embedding sharing method as a better alternative to the vanilla token embedding sharing used in the electra. the sas (self-augmentation strategy) is proposed by @xcite in order to improve electra's pre-training efficiency from the perspective of data augmentation. the sas uses a single network to jointly conduct mlm and rtd tasks in order to reduce computation cost and regularize the model parameters for training balance. essentially, the generator and the discriminator share all their transformer layers in the sas, and only two separate light-weight heads (mlm and rtd heads) are built on top of the common heavy-weight transformer layers. the mlm head also samples one token in each selected position in order to generate the corrupted input used for the next epoch of the pre-training. the sas is empirically shown by @xcite to outperform the electra in small models in glue tasks given the same computation cost, but such an advantage vanishes in larger models. the teams is proposed by @xcite to improve the electra by adding a multi-word selection (mws) task along with the original rtd task. similar to the mc-bert, the mws task, which is a multi-choice cloze test, is conducted to choose one correct answer from a candidate set of tokens provided from the generator. different from the mc-bert, however, the candidate set in the teams does not contain a reject option, since the mws task is only performed at the masked positions (instead of all positions). besides adding the mws task, the teams introduces two refinements to model structure. one is to share bottom transformer layers of the generator and the discriminator, the other is to use separate top transformer layers for rtd head and mws head. both refinements have been empirically shown to be able to further improve the performance of the teams. recently, @xcite conduct a comprehensive empirical study of electra-style pretraining techniques, and propose a corresponding pre-training recipe for model-generated denoising training objective (metro). their pre-training recipe incorporates a set of techniques to improve the efficiency and stability of large scale model pretraining, such as the zero optimizer @xcite , scaled initialization techniques, customized fused operations in mix-precision training. in terms of pre-training tasks, however, the empirical study by @xcite shows that many previously proposed tasks, such as multi-choice cloze test @xcite , clm and scl @xcite , do not provide much improvement for the rtd task in glue and squad tasks. ## method in this section, we describe our peer method, which extends the binary discriminator of the electra into a ranker. our peer method jointly trains two neutral networks, an auxiliary generator network g and a main ranker network r. each network is mainly a transformer encoder @xcite , which transforms an input token sequence ## conclusion and future work we propose the peer by extending electra's rtd task to a token quality ranking (tqr) task in order to further improve the pre-training efficiency. besides detecting whether every token is replaced or not, the tqr task also needs to rank replaced tokens into different levels according to their quality given the context. we design a scheme to retrieve rank label information from the generator so that the complete tqr task can be performed for a majority of replaced tokens. we empirically show that our proposed peer outperforms the state-ofthe-art pre-training efficient competitors in small and base scale models using the same or less computation cost. in the future, we will validate the advantages of our peer in larger scale models when sufficient computation resources are available. we also plan to improve our rank label retrieving scheme so that even larger proportion of replaced tokens can be involved in the complete tqr task.