Frustratingly Simple Pretraining Alternatives to Masked Language Modeling (2021)

Abstract: Masked language modeling (MLM), a self-supervised pretraining objective, is widely used in natural language processing for learning text representations. MLM trains a model to predict a random sample of input tokens that have been replaced by a [MASK] placeholder in a multi-class setting over the entire vocabulary. When pretraining, it is common to use alongside MLM other auxiliary objectives on the token or sequence level to improve downstream performance (e.g. next sentence prediction). However, no previous work so far has attempted in examining whether other simpler linguistically intuitive or not objectives can be used standalone as main pretraining objectives. In this paper, we explore five simple pretraining objectives based on token-level classification tasks as replacements of MLM. Empirical results on GLUE and SQUAD show that our proposed methods achieve comparable or better performance to MLM using a BERT-BASE architecture. We further validate our methods using smaller models, showing that pretraining a model with 41% of the BERT-BASE’s parameters, BERT-MEDIUM results in only a 1% drop in GLUE scores with our best objective.

## introduction masked language modeling (mlm) pretraining @xcite @xcite is widely used in natural language processing (nlp) for self-supervised learning of text representations. mlm trains a model (typically a neural network) to predict a particular token that has been replaced with a [mask] placeholder given its surrounding context. @xcite first proposed mlm with an additional next sentence prediction (nsp) task (i.e. predicting whether two segments appear consecutively in the original text) to train bert. recently several studies have extended mlm, by masking a contiguous segment of the input instead of treating each token independently @xcite @xcite . @xcite reformulated mlm in xlnet, to mask out attention weights rather than input tokens, such that the input sequence is auto-regressively generated in a random order. electric @xcite addressed the expensive softmax issue of mlm using a binary classification task, where the task is to distinguish between words sampled from the original data distribution and a noise distribution, using noise-contrastive estimation. in a different direction, previous work has also developed methods to complement mlm for improving text representation learning. @xcite have explored sentence and tokenlevel auxiliary pretraining objectives, showing improvements over nsp. albert @xcite complemented mlm with a similar task that predicts whether two sentences are in correct order or swapped. electra @xcite introduced a two-stage token-level prediction task; using a mlm generator to replace input tokens and subsequently a discriminator trying to predict whether a token has been replaced or not. despite these advances, simpler linguistically motivated or not auxiliary objective tasks acting as primary pre-training objectives substituting completely mlm have not been explored. motivated by this, we propose five frustratingly simple pretraining tasks, showing that they result into models that perform competitively to mlm when pretrained for the same duration (e.g. five days) and fine-tuned in downstream tasks in glue @xcite and squad @xcite benchmarks. contributions: (1) to the best of our knowledge, this study is the first to investigate whether linguistically and non-linguistically intuitive tasks can effectively be used for pretraining ( §2). ( 2 ) we empirically demonstrate that our proposed objec-bert original: "i cooked dinner for my family." shuffle ( 𝐶 = 2) cooked i dinner for my family shuffled shuffled original original original original bert i played dinner for my the original random original original original random bert i dinner my for the original random original shuffled shuffled random played bert [mask] cooked [mask] for my family stop word content bert i [mask] dinner for my [mask] c f bert i dinner [mask] my family for [mask] cooked random ( 𝐶 = 2) shuffle + random ( 𝐶 = 3) token type ( 𝐶 = 4) first char ( 𝐶 = 29) mlm ( 𝐶 = 50265) tives are often computationally cheaper and result in better or comparable performance to mlm across different sized models ( §4). ## pretraining tasks our methodology is based on two main hypotheses: ## results performance comparison: ## discussion based on our results, there are mainly two key elements that should be considered for designing task difficulty: a pretraining task should be moderately difficult to learn in order to induce rich text representations. for example, we can assume from the results that token type was somewhat easy for a model to learn as it is a four-way classification of identifying token properties. besides, in our preliminary experiments, predicting whether a masked token is a stop word or not (masked stop word detection) also did not exhibit competitive downstream performance to mlm as the task is a lot simpler than token type. robustness: a model should always learn useful representations from "every" training sample to solve a pretraining task, regardless of the task difficulty. for instance, figures 3 to 5 in appendix d demonstrate that shuffle needs some time to start converging across all model configurations, which means the model struggled to acquire useful representations at first. in contrast, the loss for shuffle + random consistently decreases. because shuffle + random is a multi-class classification, unlike shuffle or random, we assume that it can convey richer signals to the model and help stabilize pretraining. finally, we can also assume that mlm satisfies both elements as it is a multi-class setting over the entire vocabulary and its loss consistently decreases. ## conclusions we have proposed five simple self-supervised pretraining objectives and tested their effectiveness against mlm under various model settings. we show that our best performing, manipulated word detection task, results in comparable performance to mlm in glue and squad, whilst also being significantly faster in smaller model settings. we also show that our tasks result in higher performance trained for the same number of epochs as mlm, suggesting higher data efficiency. for future work, we are interested in exploring which has the most impact in pretraining: the data or the pretraining objective? appendices a task details here, we detail our frustratingly simple pretraining objectives, which are based on token-level classification tasks and can be used on any unlabeled corpora without laborious preprocessing to obtain labels for self-supervision.