Should You Mask 15% in Masked Language Modeling? (2023)

Abstract: Masked language models (MLMs) conventionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations; this masking rate has been widely used, regardless of model sizes or masking strategies. In this work, we revisit this important choice of MLM pre-training. We first establish that 15% is not universally optimal, and larger models should adopt a higher masking rate. Specifically, we find that masking 40% outperforms 15% for BERT-large size models on GLUE and SQuAD. Interestingly, an extremely high masking rate of 80% can still preserve 95% fine-tuning performance and most of the accuracy in linguistic probing, challenging the conventional wisdom about the role of the masking rate. We then examine the interplay between masking rates and masking strategies and find that uniform masking requires a higher masking rate compared to sophisticated masking strategies such as span or PMI masking. Finally, we argue that increasing the masking rate has two distinct effects: it leads to more corruption, which makes the prediction task more difficult; it also enables more predictions, which benefits optimization. Using this framework, we revisit BERT’s 80-10-10 corruption strategy. Together, our results contribute to a better understanding of MLM pre-training.

## introduction pre-trained language models have transformed the landscape of natural language processing @xcite @xcite . they are trained on vast quantities of text data and acquire rich and versatile language representations. compared to autoregressive models, which always predict the next token in a sequence, masked language models (mlms) like bert @xcite predict a masked subset of input tokens based on the remaining context and are more effective on downstream tasks due to their bidirectional nature. bert chooses a 15% masking rate, based on the reasoning that models cannot learn good representations when too much text is masked, and the training is inefficient when too little is masked. surprisingly, this important choice has been underexplored since 15% masking is used ubiquitously by bert's successors @xcite @xcite @xcite , regardless of model sizes, masking strategies and optimization recipes. 2 in this work, we aim to understand the impact of masking rates. we hypothesize that the optimal masking rate is not universally 15%, but should depend on other factors. first, we consider the impact of model sizes and establish that indeed larger models should adopt higher masking rates ( §3). specifically, we find that under an efficient pre-training recipe @xcite , 40% outperforms 15% for bert-large size models when fine-tuning on glue and squad. interestingly, we observe that large models can still learn good representations even for very high masking rates: if we mask as much as 80% of input tokens and pre-trained models have a perplexity of more than 1000, the learned representations can still preserve more than 95% of fine-tuning performance on downstream tasks, compared to the default 15% masking ( we then focus on the strategy of which tokens to mask as an additional factor to the optimal masking rate of mlms ( §5). we find that different masking rates should be used with different masking strategies, and the default uniform masking bene- finally, we propose to dissect the masking rate into two factors ( §6): the corruption rate-how much of the context is corrupted (masked)-and the prediction rate-how much of the tokens the model predicts on. in mlms, both are set to the masking rate. however, these two factors have opposing effects: higher prediction rates generate more training signals and benefit the optimization, while higher corruption rates make the prediction task more challenging by providing less context. to study the two factors independently, we design ablation experiments to disentangle corruption and prediction rates. thus, we can verify that models benefit from higher prediction rates and suffer from more corruption. using this framework, we also discuss bert's practice of predicting on original or random tokens (the 80-10-10 rule), and we find that models usually perform worse under this corruption strategy ( §7). together, our results demonstrate the overlooked impact of the masking rate in mlm pre-training and our analysis disentangles its opposing effects of corruption and prediction. we conclude by discussing the relation to work in other models and modalities ( §8) and by highlighting several new avenues for efficient mlm in the future ( §9). ## mlms in high-masking regimes the success of masking 40% over 15% motivates us to explore what happens at even larger masking rates. therefore, we pre-train additional large models with masking rates of up to 80%. we consider the question of what representations an mlm can learn with such limited input as the last masked sentence in mlms learn with extreme masking. we first confirm in figure 4 : evaluating our models on the blimp benchmark @xcite using pseudo loglikelihood scoring @xcite . we hypothesize that mlms at such high masking rates may be understood as a powerful skipgram model @xcite , e.g., masking 80% of a 128 token sequence still learns skip-grams of length up to 26. furthermore, when compared to the simple word2vec model, our transformer models have access to positional information for each context token and prediction. figure 6: higher masking rates increase the probability that an entire pmi span is masked (left) under different masking strategies. uniform masking with a 40% rate masks as many pmi spans as regular pmi masking at 15%. masks form longer spans for higher masking rates in uniform sampling, while the average length is fixed at 3 for t5-style span masking (which cannot be enforced for very high masking rates). rates are higher than 15%; (2) the optimal masking rates for span masking and pmi masking are lower than that of uniform masking; (3) when all strategies adopt the optimal masking rates, the uniform masking achieves similar and even better results compared to the advanced strategies. we also remark that, when masking with 15%, simply increasing the masking rate can be a more effective way to increase performance on squad than switching from uniform masking to another more advanced strategy. more fine-grained results with these masking strategies are included in appendix e. ## understanding masking as corruption and prediction in this section, we analyze how masking rates affect the pre-training process of mlms, through two distinct perspectives: task difficulty and optimization. we identify that the masking rate m determines two import aspects of the pre-training problem: the corruption rate m corr and the prediction rate m pred . m corr is the proportion of tokens that are erased from the input sequence-typically by substituting @xcite . m pred is the proportion of tokens that the models predict, and each of those tokens contributes to the cross-entropy loss. in eq. ( 1 ), m corr controls how much content is corrupted in x compared to the original sentence x, and m pred controls the number of predictions in the set m. usually, both the corruption and the prediction rates are tied to the masking rate, i.e., m @xmath0, but they may impact representation quality differently. m corr controls task difficulty. masked language modeling attempts to learn a conditional probability distribution over the vocabulary given the corrupted context p(• | x) during pre-training. if a larger proportion of the input is corrupted, a token prediction is conditioned on fewer context tokens, making predictions harder and more uncertain. m pred affects optimization. predicting more means the model learns from more training signals, so higher prediction rates boost the model performance. from another perspective, each prediction at each masked token leads to a loss gradient, which is averaged to optimize the weights of the model. averaging across more predictions has a similar effect to increasing the batch size, which is proved to be beneficial for pre-training @xcite . experiments. in masked language modeling, both m corr and m pred are determined by the overall masking rate. to study how m corr and m pred affect the downstream performance independently, we design a simple ablation experiment to disentangle them: the ablation shows that when we tune the masking rate, we are tuning the corruption rate and the prediction rate together, which have antagonistic effects. the final outcome is decided by which rate weighs more-the model benefits from higher masking rates if the hindrance brought by high corruption is surpassed by the advantage from predicting more. many factors may affect the balance between the two-for example, model sizes and masking strategies as we discussed in §3 and §5. ## conclusion & discussion in this work, we conduct a comprehensive study on the masking rates of mlms. we discover that 15% is not universally optimal, and larger models should adopt a higher masking rate. we also find that masking strategies should be considered together with masking rates, and uniform masking needs a higher masking rate than more sophisticated masking strategies. we gain a better understanding of masking rates by disentangling them as corruption rates and prediction rates and analyze the 80-10-10 corruption strategy that are widely used in bert models. based on our findings, we discuss the implications of high masking rates and future directions of efficient mlm pre-training: implications on higher masking rates. a direct takeaway from our findings is that larger models may adopt higher masking rates for better sample efficiency. figure 1 shows that a large model with 40% masking can achieve comparable results to a 15% baseline on several tasks with half the training time. larger models also exhibit faster convergence for a given computational budget: @xcite suggest it is more efficient to train larger models for fewer steps, as opposed to training smaller models for longer. this can be combined with higher masking rates for better sample efficiency. separating masked and unmasked tokens. the training efficiency can potentially benefit from encoding masked and unmasked tokens separately, where masked tokens use a much lighter-weight module. if a high masking rate is taken, this can significantly reduce the training cost due to the shorter input to the encoder. a similar approach has been explored by masked autoencoders in vision @xcite , where 75% of the input patches are masked and removed from the input of the heavy encoder to achieve a 4.1× speedup. recently, @xcite have applied these architectural improvements to natural language pre-training, and together with a high masking rate can accelerate mlm by a third of the pre-training budget. disentangling corruption and prediction. models perform better when trained with lower corruption rates and higher prediction rates. however, in standard mlms, those two factors are always tied to the masking rate. methods which can encode a sequence once and then efficiently predict many small sets of masks, for example by manipulating the attention, could substantially accelerate masked language modeling pre-training.