Learning Better Masking for Better Language Model Pre-training (2023)

Abstract: Masked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training. However, the model may receive complicated impact from pre-training status, which changes accordingly as training time goes on. In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings. We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks. Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pre-training.

## introduction pre-trained language models (prlms) have played an essential role in many natural language processing tasks @xcite @xcite @xcite . generally speaking, prlms can be seen as an automatic denoising encoder and may be conveniently obtained through a self-supervised learning way. masked language modeling (mlm) pioneered by bert @xcite is a widely used denoising method for language model pre-training @xcite . in mlm pre-training, a subset of tokens in a sequence is masked with a certain masking ratio, and the masked sequence is fed to the prlm, which is required to predict the masked tokens. masking in mlm is a process in terms of sampling masked tokens from a huge data space to generate training batches, in which mlm may be heavily controlled by two main factors, masking ratio and masked contents. so far, only a few studies have ever considered optimal settings for better mlm from quite limited perspectives. especially, all known works only take time-invariant mlm setting into account despite the huge time variance of the model during a lengthy pre-training. for example, carefully considered masked units like ngram, entity and span @xcite @xcite ) are adopted throughout the entire pre-training. another example is that exploring a good enough (but still fixed) masking ratio has also been considered in @xcite . given the circumstances that time-invariant masking applied in most mlm is not adaptive to the changeable process of language model pre-training, time-invariant setting hardly hopefully reaches an optimal outcome. this motivates us to explore the influence of masking ratio and masked content in mlm pre-training and propose time-variance mlm setting to verify our hypothesis for better prlms. we first explore the influence of different masking ratios on downstream tasks at different stages throughout the entire pre-training instead of the only final stage. we find that a high masking ratio gives better performance for downstream tasks in early stage, while a low ratio has a faster training speed. thus we choose a higher ratio as the starting point and decay the masking ratio to a lower value during the pre-training, namely masking ratio decay (mrd), which can significantly outperform the performance of the fixed ratio. mrd indicates that mlm benefits from a time-variant masking ratio at different training stages. to handle training maturity for different types of words, we propose pos-tagging weighted (ptw) masking to adaptively adjust the masking probabilities of different types of words according to the current training state. ptw masking makes the model have more chance to learn 'difficult' types of words and less chance for 'easy' ones from the perspective of part-of-speech. by introducing this adaptive schedule, our experimental results show that mlm benefits from learning mostly nonfunction words. our contributions are three folds: 1) we analyze the insufficiency of current masking strategies from the perspectives of masking ratio and masked content and give a better understanding of mlm pre-training in terms of masking. 2) to our best knowledge, this is a pioneer study to analyze the impact of time-variant masking both in masking ratio and masked content in mlm pre-training. 3) our analysis shows that the time-variant masking schedules can significantly improve training efficiency and effectiveness. our sources will be publicly available. ## preliminary experiments this section presents our preliminary experiments that motivate us to explore time-variant masking schedules. we train bert-base @xcite with the widely-used english wikipedia corpus to observe the influence of masking during pre-training by measuring the downstream performance on the squad v1.1 dataset @xcite (more experimental details will be given in section 4). the experiments aim to study how the language model learns from the masked tokens when using conventional random-token masking from the perspectives of the masking ratio and masked content. ## time-variant masking strategies in this section, we will present our exploration of time-variant masking on masking ratio and masked content inspired by our findings above. the overview of our time-variant masking is presented in figure 3 . ## related work the pre-processing of the mlm is to replace a subset of the tokens in the input with [mask] tokens, which has two considerations to optimize: how many tokens to mask (masking ratio) and what tokens to mask (masked content). ## conclusion masked language model pre-training can be generally defined by two main factors, masking ratio and masked contents. the random-token masking scheme adopted by existing studies treats all words equally and maintains a fixed ratio throughout the entire pre-training, which has been shown suboptimal in our analysis. to better unleash the strength of mlm, we explore two kinds of time-variant masking strategies, namely, masking ratio decay (mrd) and pos-tagging weighted (ptw) masking. experimental results verify our hypothesis that mlm benefits from time-variant setting both in masking ratio and masked content according to dynamic training states. our further analysis show that these two time-variant masking schedules greatly improve pre-training efficiency and the performance of downstream tasks.