Self-Evolution Learning for Discriminative Language Model Pretraining (2023)

Abstract: Masked language modeling, widely used in discriminative language model (e.g., BERT) pretraining, commonly adopts a random masking strategy. However, random masking does not consider the importance of the different words in the sentence meaning, where some of them are more worthy to be predicted. Therefore, various masking strategies (e.g., entity-level masking) are proposed, but most of them require expensive prior knowledge and generally train from scratch without reusing existing model weights. In this paper, we present Self-Evolution learning (SE), a simple and effective token masking and learning method to fully and wisely exploit the knowledge from data. SE focuses on learning the informative yet under-explored tokens and adaptively regularizes the training by introducing a novel Token-specific Label Smoothing approach. Experiments on 10 tasks show that our SE brings consistent and significant improvements (+1.43 2.12 average scores) upon different PLMs. In-depth analyses demonstrate that SE improves linguistic knowledge learning and generalization.

## introduction masked language modeling (mlm), which commonly adopts a random masking strategy to select the mask tokens, has become the de-facto standard for discriminative pretrained language models (plms) @xcite @xcite . however, such a random masking process is usually criticized as being sub-optimal, as it allocates an equal masking rate for all tokens. in particular, the masked tokens are sometimes too easy to guess with only local cues or shallow patterns @xcite , while the informative tokens that carry more critical linguistic knowledge may be neglected @xcite . for example, "bush" and "sharon" express more important meaning than "a" in the sample sentence "bush held a talk with sharon". mlm with predicting the above easy-to-guess tokens, e.g., "a", would lead to low data efficiency and sub-optimal model capability. to address this problem, various methods have been carefully designed to improve mlm via fully leveraging the training data @xcite @xcite . the common goal is to inject language prior knowledge into the pretraining process @xcite . although empirically successful, there are still some limitations. first, they usually require annotation derived from off-the-shelf tools to select mask tokens, which is not only expensive but also too deterministic foot_0 , and may cause error propagation from the third-party tool. for instance, @xcite employ external linguistic tools, e.g., stanford corenlp @xcite , to annotate the entities. second, to ensure the effectiveness of the masking strategy, most previous works train plm from scratch without reusing the existing models trained with vanilla mlm @xcite @xcite , which is wasteful and inefficient. thus, there raises a question: whether we can strengthen the plm capability and data efficiency through further learning from the informative yet under-explored tokens, where such tokens are determined by the existing plm itself. in fact, an off-the-shelf plm already has the ability to determine the worthy and informative tokens that should be further exploited, as the representation of plm generally can reveal good enough linguistic properties @xcite . for example, tokens that plms predict incorrect or low confidence are usually more hard-to-learn and challenging, which are essential for further training. also, the conjecture to improve the off-the-shelf plm is model-agnostic, green, and efficient, thus having the great potential to evolve any existing discriminative plms. motivated by this, we design a simple and effective self-evolution learning (se) mechanism to improve the pretraining of discriminative plms. specifically, the se contains two stages: ❶selfquestioning and ❷self-evolution training. in stage 1, the plm is forced to locate the informative but under-explored tokens foot_1 from the pretraining data. after locating these hard-to-learn tokens, we then encourage the plm to learn from them in stage 2, where we basically follow the vanilla mlm to mask these tokens and then optimize the plm by minimizing the loss between the predictions and one-hot labels. it should be noted that due to the hard-to-learn properties, directly enforcing the plm to fit the hard labels may lead to overfitting or overconfidence problem @xcite . inspired by the label smoothing (ls) @xcite that regularizes the learning by smoothing target labels with a pre-defined (static) prior distribution, we propose a novel token-specific label smoothing (tls) approach. our tls considers both the precise hard label and, importantly, the easily-digestible foot_2 distribution that is adaptively generated by the plm itself. we validated our se on several benchmarks including glue @xcite , super-glue @xcite , squad2.0 @xcite , swag @xcite and lama @xcite over several plms: bret @xcite )-base, -large, roberta @xcite -base, and -large. experiments demonstrate the effectiveness and universality of our approach. extensive analyses confirm that se effectively enhances the ability of plms on linguistic knowledge learning, model generalization and robustness. contributions our main contributions are: ## related works in recent years, we have witnessed numerous discriminative plms @xcite @xcite @xcite ) that achieved tremendous success in various natural language understanding (nlu) tasks. although the discriminative plms vary in terms of pretraining data or model architecture, they are commonly based on mlm loss function. mlm mechanism is pioneered in bert @xcite that uses a random masking strategy to mask some tokens, and then enforces the plm to learn to recover word information from the masked tokens. obviously, the vanilla mlm is a linguisticagnostic task, as the random masking procedure does not integrate linguistic knowledge explicitly, which is sub-optimal. thus, several previous studies attempt to improve mlm by exploring a diverse of linguistically-motivated masking strategies, such as entity-level masking @xcite , spanlevel masking @xcite , n-grams masking @xcite , etc., to fully leverage the pretraining data. although achieving remarkable performance, these strategies still have some limitations. first, their implementations are relatively complex, as they usually require annotation derived from external models or tools to select tokens for masking. even for the unsupervised pmi-masking @xcite , it is still expensive to measure the pointwise mutual information for pretrain-level large-scale data, and the annotated labels are static, while our se could obtain dynamic annotations via given existing plms. second, in order to ensure the effectiveness of masking strategy, most previous works @xcite @xcite train the language models from scratch without reusing the existing plms trained with vanilla mlm, which is wasteful and inefficient. along the same research line, in this paper, we improve the mlm-based plms with a novel selfevolution learning mechanism. instead of training a plm from scratch based on a carefully-designed and complex masking strategy, our mechanism aims to strengthen the plm's capability and data efficiency by further learning from the informative yet under-explored tokens, which are determined by the existing plm itself. ## discussion to better understand se, we conduct extensive analyses to discuss whether it gains better generalization/ robustness and knowledge-learning ability. ## conclusion in this paper, we propose a simple and effective selfevolution (se) learning mechanism to improve the existing discriminative plms by fully exploiting the knowledge from data. se follows two stages, i.e., self-questioning and self-evolution training, and can be used to evolve any mlm-based plms with a simple recipe: continue pretraining with se. we empirically demonstrated the effectiveness and universality of the se on a series of widelyused benchmarks. further analyses show our approach improves the generalization, robustness, and knowledge-learning ability. we hope our work could facilitate more research on how to improve existing trained models after all the previous plm weights are expensive and knowledgeable.