How does the pre-training objective affect what large language models learn about linguistic properties? (2022)

Abstract: Several pre-training objectives, such as masked language modeling (MLM), have been proposed to pre-train language models (e.g. BERT) with the aim of learning better language representations. However, to the best of our knowledge, no previous work so far has investigated how different pre-training objectives affect what BERT learns about linguistics properties. We hypothesize that linguistically motivated objectives such as MLM should help BERT to acquire better linguistic knowledge compared to other non-linguistically motivated objectives that are not intuitive or hard for humans to guess the association between the input and the label to be predicted. To this end, we pre-train BERT with two linguistically motivated objectives and three non-linguistically motivated ones. We then probe for linguistic characteristics encoded in the representation of the resulting models. We find strong evidence that there are only small differences in probing performance between the representations learned by the two different types of objectives. These surprising results question the dominant narrative of linguistically informed pre-training.

## introduction the most popular way to pre-train a transformerbased @xcite language model (lm), e.g. bert @xcite , is by optimizing a masked language modeling (mlm) objective. the mlm task was inspired by the cloze task @xcite , where humans were asked to guess omitted words in a sentence using its context, knowledge of syntax and other skills. the premise is that such an objective will guide a lm to encode linguistic information. apart from mlm, different types of objectives have been recently proposed. @xcite introduced a pre-training objective based on token order permutations. @xcite proposed a replaced token detection pre-training task, that uses the output of a small mlm to corrupt the input by replacing some of the tokens. it then trains a discriminative model to predict if a token has been replaced or not. @xcite explored various sentence and token-level auxiliary pre-training tasks (e.g. sentence ordering, term-frequency prediction), as better alternatives to the next sentence prediction (nsp) auxiliary task originally used to train bert. @xcite introduced the sentence-order prediction task that focuses on the inter-sentence coherence, by predicting if two contiguous sentences have been swapped or not. @xcite proposed another intersentence pre-training task, that helps lms to encode discourse relationships between sentences using contrastive learning. @xcite showed that a non-linguistically intuitive task (i.e. masked first character prediction) can effectively be used for pre-training. meanwhile, several studies have explored how well and to what extent lms learn linguistic information. this is usually examined using probing tasks, i.e. simple classification tasks that test the lm's encodings for a single linguistic feature such as grammatical information. it has been found through probing that bert encodes syntactic @xcite @xcite @xcite and semantic information @xcite @xcite . however, hall @xcite argue that bert's syntactic abilities may have been overestimated. in this paper, we hypothesize that linguistically motivated objectives (e.g. mlm) should help bert to acquire better linguistic knowledge compared to using non-linguistically motivated objectives, i.e. tasks that are hard for humans to guess the association between the input and the label to be predicted. to this end, we seek to answer the following research question: how does the pretraining objective affect what lms learn about the english language? our findings challenge the mlm status quo, showing that pre-training with non-linguistically informative objectives ( §2) results in models with comparable linguistic capabilities, as measured by standard probing benchmarks ( §3). these surprising results ( §4) suggest that careful analysis of how lms learn is critical to further improve language modeling ( §5). ## pre-training objectives we experiment with five different pre-training objectives. two of them are considered linguistically motivated while the rest are not. ## probing tasks probing tasks @xcite @xcite are used to explore in what extent linguistic properties are captured by lms. a model is normally trained, using the representations of a language model, to predict a specific linguistic property. if it achieves high accuracy, it implies that the lm encodes that linguistic property. in this work, we use nine standard probing tasks introduced by @xcite to examine the representation output for each layer of the different lms we pre-train following @xcite . these tasks probe for surface, syntactic and semantic information. the dataset for each probing task contains 100k sentences for training, 10k sentences for validation and another 10k sentences for testing. 2 we train a multi-layer perceptron (mlp) classifier for each probing task using the recommended hyperparameters in the senteval toolkit @xcite . surface information task: sentlen aims for correctly predicting the number of words in a sentence. ## discussion theoretically, lms with non-linguistically motivated objectives would be expected to perform drastically worse than lms pre-trained using mlm in both downstream tasks and linguistic capabilities. however, our results show that both types of lms have surprisingly close performance (after fine-tuning on downstream tasks) and linguistic capabilities (after probing them) using the same training data, architecture and training scheme. we speculate that the pre-training data, and the size of model sentlen treedepth topconst bshift tense subjnum objnum somo coordinv (surface) (syntactic) (syntactic) (syntactic) (semantic) (semantic) (semantic) (semantic) (semantic) medium -250k steps pre-training mlm 92 the models have more impact on the effectiveness of lms than the pre-training objectives. furthermore, the comparable performance of different objectives in probing suggests that lms mainly learn word co-occurrence information from pre-training @xcite and that the objectives may have a little effect to what actually learn about linguistic properties. recent studies have explored the limitations of using probing tasks to draw conclusions over a model's linguistic knowledge with some also suggesting improvements or alternative probing methods @xcite @xcite . however, our results show no substantial differences in the performance across tasks that probe for syntactic or semantic information between models that have been pre-trained using linguistically motivated objectives or non-linguistically motivated ones. ## conclusions in this work, we compared the linguistic capabilities of lms. surprisingly, our results show that pretraining with linguistically motivated objectives obtain comparable performance to non-linguistically motivated objectives. this suggests that the data and the size of the model could be more influential than the objectives themselves in language model-ing. in future work, we plan to extend our experiments into other languages and probing tasks.