How does the task complexity of masked pretraining objectives affect downstream performance? (2023)

Abstract: Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts. Although simpler and computationally efficient pretraining objectives, e.g., predicting the first character of a masked token, have recently shown comparable results to MLM, no objectives with a masking scheme actually outperform it in downstream tasks. Motivated by the assumption that their lack of complexity plays a vital role in the degradation, we validate whether more complex masked objectives can achieve better results and investigate how much complexity they should have to perform comparably to MLM. Our results using GLUE, SQuAD, and Universal Dependencies benchmarks demonstrate that more complicated objectives tend to show better downstream results with at least half of the MLM complexity needed to perform comparably to MLM. Finally, we discuss how we should pretrain a model using a masked objective from the task complexity perspective.

## introduction masked language modeling (mlm) @xcite , where a model needs to predict a particular token that is replaced with a mask placeholder given its surrounding context, is a widely used self-supervised pretraining objective in natural language processing. recently, simpler pretraining objectives have shown promising results on downstream tasks. @xcite have proposed various token-level and sentencelevel auxiliary pretraining objectives, showing improvements over bert @xcite . @xcite and @xcite have demonstrated that such token-level ob-* equal contribution 1 our code and pretrained models are available at https: //github.com/hitachi-nlp/mlm-probe-acl2023 . although these simple token-level objectives themselves, e.g., predicting the first character of a masked token (first char) @xcite , have exhibited competitive downstream performances to mlm with smaller computations, no objectives using mask tokens are not clearly comparable to mlm on downstream tasks. we conjecture that the main reason behind the performance difference lies in its lack of complexity, i.e., the number of classes to be predicted, and similar arguments have been made for auxiliary task ineffectiveness @xcite and pretraining task design @xcite . this paper sheds light on the task complexity of masked pretraining objectives and investigates rq1: whether a more complex objective, becoming closer to mlm, can achieve a better downstream result and rq2: how much complexity they need to obtain comparable results to mlm. to this end, we propose masked n character prediction as a control task, which requires us to predict the first or last n characters of a masked token, allowing us to empirically evaluate how the task complexity affects downstream performance by varying n. we pretrain 14 different types of models with the proposed control task in addition to mlm for reference and evaluate their downstream performance on the glue @xcite , squad @xcite , and universal dependencies (ud) @xcite benchmarks. we also conduct a cost-benefit analysis of performance gains with respect to task complexity and analyze how to select an optimal complexity for a given task. contributions (1) we model the task complexity of a masked pretraining objective as masked n character prediction ( §2) and revealed how it affects downstream performance ( §4). (2) we conduct two analyses to provide insights into how we should fi rs t | la st c ha r 1 c ha r 2 c ha rs 3 c ha rs 4 c ha rs 5 c ha rs 9 c ha rs m lm 10 foot_0 10 foot_1 10 foot_2 first last pretrain a model by using a masked objective from the task complexity perspective ( §5). ## methodology our main hypothesis is that the more complexity, i.e., the number of classes to be predicted, masked pretraining objectives have, the better the downstream performance they will achieve. this is because a more complex task should have a larger number of classes to be predicted, giving more informative signals to a model via training. to verify the hypothesis empirically and answer the two research questions listed in §1, we extend first char @xcite and let a model predict the first or last @xmath0. the task is trained with the token-level cross-entropy loss averaged over masked tokens. our extension allows us to evaluate how the complexity affects downstream performance by varying n. figure 1 shows the number of classes to be predicted for n chars when using a pretrained tokenizer of roberta @xcite . we can see that the larger n, the closer the objective becomes to mlm. for 1 char, we simply pick up the first character of each token in the vocabulary instead of casting it into 29 classes as in first char 3 , resulting in 256 types of characters. ## experimental setup here, we describe our experimental setups for both pretraining and fine-tuning. 4 model we used the base configuration of bert @xcite . the model consists of 12 hidden layers and attention heads, and the dimensions of hidden layers and intermediate feed-forward layers are 768 and 3072, respectively. we simply put a linear layer on the bert model for n chars and first char. baselines we pretrained mlm and first char for reference to evaluate the influence of masked pretraining objective complexity. we also set up last char, where a model needs to predict the last character of a masked token from 29 categories the same as in first char. following devlin et al. (2019) , we pretrained all models on english wikipedia and bookcorpus @xcite using the datasets @xcite library. we set the maximum sequence length to 512. we tokenized texts using byte-level byte-pair-encoding @xcite , and the resulting corpora consist of 10 million samples and 4.9 billion tokens in total. ## conclusion this paper analyzed the impact of masked objective complexities over downstream performance, motivated by the assumption that the lack of task complexity in simple masked pretraining objectives (e.g., first char) affects the performance degradation compared to mlm. experiments using the glue, squad, and ud datasets revealed that the task complexity significantly affected downstream performance with at least 35.7% of the mlm prediction classes needed to perform comparably to mlm on at least one of the high-resource corpora. our analysis also showed that there exists a tradeoff between downstream performance and computational efficiency, and different downstream tasks might have different optimal complexities. future work includes analyzing other properties (e.g., fairness) with respect to task complexity.