Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021)

Abstract: A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent in classical NLP pipelines. In this paper, we propose a different explanation: MLMs succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. To demonstrate this, we pre-train MLMs on sentences with randomly shuffled word order, and show that these models still achieve high accuracy after fine-tuning on many downstream tasks—including tasks specifically designed to be challenging for models that ignore word order. Our models perform surprisingly well according to some parametric syntactic probes, indicating possible deficiencies in how we test representations for syntactic information. Overall, our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.

## introduction the field of natural language processing (nlp) has become dominated by the pretrain-and-finetune paradigm, where we first obtain a good parametric prior in order to subsequently model downstream tasks accurately. in particular, masked language model (mlm) pre-training, as epitomized by bert @xcite , has proven wildly successful, although the precise reason for this success has remained unclear. on one hand, we can view bert as the newest in a long line of nlp techniques @xcite @xcite @xcite that exploit the wellknown distributional hypothesis @xcite . 1 on the other hand, it has been claimed that bert 1 one might even argue that bert is not actually all that different from earlier distributional models like word2vec @xcite , see appendix a. in this work, we aim to uncover how much of mlm's success comes from learning simple distributional information, as opposed to grammatical abstractions @xcite . we disentangle these two hypotheses by measuring the effect of removing word order information during pre-training: any sophisticated (english) nlp pipeline would presumably depend on the syntactic information conveyed by the order of words. surprisingly, we find that most of mlm's high performance can in fact be explained by the "distributional prior" rather than its ability to replicate the classical nlp pipeline. concretely, we pre-train mlms (roberta, @xcite ) on various corpora with permuted word order while preserving some degree of distributional information, and examine their downstream performance. we also experiment with training mlms without positional embeddings, making them entirely order agnostic, and with training on a corpus sampled from the source corpus's unigram distribution. we then evaluate these "permuted" models in a wide range of settings and compare with regularly-pre-trained models. we demonstrate that pre-training on permuted data has surprisingly little effect on downstream task performance after fine-tuning (on non-shuffled training data). it has recently been found that mlms are quite robust to permuting downstream test data @xcite @xcite and even do quite well using permuted "unnatural" downstream train data @xcite . here, we show that downstream performance for "unnatural language pre-training" is much closer to standard mlm pretraining than one might expect. in an effort to shed light on these findings, we experiment with various probing tasks. we verify via non-parametric probes that the permutations do in fact make the model worse at syntax-dependent tasks. however, just like on the downstream finetuning tasks, permuted models perform well on parametric syntactic probes, in some cases almost matching the unpermuted model's performance, which is quite surprising given how important word order is crosslinguistically @xcite @xcite . our results can be interpreted in different ways. one could argue that our downstream and probing tasks are flawed, and that we need to examine models with examples that truly test strong generalization and compositionality. alternatively, one could argue that prior works have overstated the dependence of human language understanding on word order, and that human language understanding depends less on the structure of the sentence and more on the structure of the world, which can be inferred to a large extent from distributional information. this work is meant to deepen our understanding of mlm pre-training and, through this, move us closer to finding out what is actually required for adequately modelling natural language. more recently, several research groups have investigated information order for words rather than sentences as an evaluation of model humanlikeness. @xcite investigate the task of natural language inference (nli) and find high accuracy on permuted examples for different transformer and pre-transformer era models, across english and chinese datasets @xcite . @xcite use targeted permutations on robertabased models and show word order insensitivity across natural language inference (mnli), paraphrase detection (qqp) and sentiment analysis tasks (sst-2). @xcite show insensitivity on a larger set of tasks, including the entire glue benchmark, and find that certain tasks in glue, such as cola and rte are more sensitive to per-mutations than others. @xcite recently observed that bert accuracy decreases for some word order perturbed examples, but not for others. in all these prior works, models were given access to normal word order at (pre-)training time, but not at test-time or (sometimes) fine-tuning time. it was not clear whether the model acquires enough information about word order during the fine-tuning step, or whether it is ingrained in the pre-trained model. in this work, we take these investigations a step further: we show that the word order information needed for downstream tasks does not need to be provided to the model during pre-training. since models can learn whatever word order information they do need largely from fine-tuning alone, this likely suggests that our downstream tasks don't actually require much complex word order information in the first place (cf., glavaš and vulić 2021). randomization ablations. random controls have been explored in a variety of prior work. @xcite show that random sentence encoders are surprisingly powerful baselines. gauthier and @xcite use random sentence reordering to label some tasks as "syntax-light" making them more easily decodeable from images of the brain. @xcite show that entire layers of mlm transformers can be randomly initialized and kept frozen throughout training without detrimental effect and that those layers perform better on some probing tasks than their frozen counterparts. models have been found to be surprisingly robust to randomizing or cutting syntactic tree structures they were hoped to rely on @xcite , and randomly permuting attention weights often induces only minimal changes in output @xcite . in computer vision, it is well known that certain architectures constitute good "deep image priors" for fine-tuning @xcite or pruning @xcite , and that even randomly wired networks can perform well at image recognition @xcite . here, we explore randomizing the data, rather than the model, to assess whether certain claims about which phenomena the model has learned are established in fact. synthetic pre-training. @xcite found that pre-training on synthetically generated fractals for image classification is a very strong prior for subsequent fine-tuning on real image data. in language modeling, papadimitriou and jurafsky (2020) train lstms @xcite on non-linguistic data with latent structure such as midi music or java code provides better test performance on downstream tasks than a randomly initialized model. they observe that even when there is no vocabulary overlap among source and target languages, lstm language models leverage the latent hierarchical structure of the input to obtain better performance than a random, zipfian corpus of the same vocabulary. on the utility of probing tasks. many recent papers provide compelling evidence that bert contains a surprising amount of syntax, semantics, and world knowledge @xcite @xcite @xcite . many of these works involve diagnostic classifiers @xcite or parametric probes, i.e. a function atop learned representations that is optimized to find linguistic information. how well the probe learns a given signal can be seen as a proxy for linguistic knowledge encoded in the representations. however, the community is divided on many aspects of probing @xcite including how complex probes should be. many prefer simple linear probes over the complex ones @xcite @xcite . however, complex probes with strong representational capacity are able to extract the most information from representations @xcite @xcite . here, we follow pimentel et al. (2020a) and use both simple (linear) and complex (non-linear) models, as well as "complex" tasks (dependency parsing). as an alternative to parametric probes, stimulus-based non-parametric probing @xcite @xcite @xcite @xcite has been used to show that even without a learned probe, bert can predict syntactic properties with high confidence @xcite . we use this class of non-parametric probes to investigate roberta's ability to learn word order during pre-training. ## approach we first describe the data generation and evaluation methodology used in this paper. we use the roberta (base) @xcite mlm architecture, due to its relative computational efficiency and good downstream task performance. we ex-pect that other variants of mlms would provide similar insights, given their similar characteristics. ## downstream task results in this section, we present the downstream task performance of the models defined in §3. for evaluation, we report matthews correlation for cola and accuracy for all other tasks. ## probing results to investigate how much syntactic information is contained in the mlm representations, we evaluate several probing tasks on our trained models. we consider two classes of probes: parametric probes, which make use of learnable parameters, and non-parametric probes, which directly exam-ine the language model's predictions. ## discussion the assumption that word order information is crucial for any classical nlp pipeline (especially for english) is deeply ingrained in our understanding of syntax itself @xcite : without order, many linguistic constructs are undefined. our fine-tuning results in §4.1 and parametric probing results in §5.1, however, suggests that mlms do not need to rely much on word order to achieve high accuracy, bringing into question previous claims that they learn a "classical nlp pipeline." one might ask, though, whether an nlp pipeline would really need natural word order at all: can transformers not simply learn what the correct word order is from unordered text? first, the lower nonparametric probing accuracies of the randomized models indicate that they are not able to accurately reconstruct the original word order (see also appendix d). but even if models were able to "unshuffle" the words under our unnatural pre-training set up, they would only be doing so based on distributional information. models would then abductively learn only the most likely word order. while models might infer a distribution over possible orders and use that information to structure their representations @xcite , syntax is not about possible or even the most likely orders: it is about the actual order. that is, even if one concludes in the end that transformers are able to perform word order reconstruction based on distributional information, and recover almost all downstream performance based solely on that, we ought to be a lot more careful when making claims about what our evaluation datasets are telling us. thus, our results seem to suggest that we may need to revisit what we mean by "linguistic structure," and perhaps subsequently acknowledge that we may not need human-like linguistic abilities for most nlp tasks. or, our results can be interpreted as evidence that we need to develop more challenging and more comprehensive evaluations, if we genuinely want to measure linguistic abilities, however those are defined, in nlp models. there are many interesting and potentially exciting avenues for future work that we could not explore due to limitation of space. an interesting question revolves around whether this phenomenon is more pronounced for english than for other languages. it is natural to wonder whether more word-order flexible or morphologically-rich languages would suffer from the same problem. using the methods discussed in this work, we could imagine devising a way to determine the degree of order-dependence for tasks across languages. another possible extension pertains to other tasks, including extractive question answering (qa) or sequence tagging, for which we can also to deter-mine whether word order information is acquired downstream or during pre-training. the sensitivity of generative models to word order permuted input could also be investigated further. recent work by @xcite begins this discussion, by showing that a machine translation (mt) model can often arrive at the gold source translation when provided with input sentences that have had their words permuted using parse trees. relatedly, @xcite also investigates targeted parse-tree-based perturbations as a means of evaluating model robustness. o'connor and andreas (2021) also demonstrate the insensitivity of transformers towards syntax manipulations while achieving low perplexity in language modeling tasks. exploring model sensitivity to word order permutations for approaches that unify generation and classification (e.g., multitasking) could also be interesting future work. ## conclusion in this work, we revisited the hypothesis that masked language modelling's impressive performance can be explained in part by its ability to learn classical nlp pipelines. we investigated targeted pre-training on sentences with various degrees of randomization in their word order, and observed overwhelmingly that mlm's success is most likely not due to its ability to discover syntactic and semantic mechanisms necessary for a traditional language processing pipeline during pre-training. instead, our experiments suggest that mlm's success can largely be explained by it having learned higher-order distributional statistics that make for a useful prior for subsequent fine-tuning. these results should hopefully encourage the development of better, more challenging tasks that require sophisticated reasoning, and harder probes to narrow down what exact linguistic information is present in the representations learned by our models. methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 529-539, hong kong, china. association for computational linguistics. danilo giampiccolo, bernardo magnini, ido dagan, and william b dolan. 2007. the third pascal recognizing textual entailment challenge. in proceedings of the acl-pascal workshop on textual entailment and paraphrasing, pages 1-9. mario giulianelli, jack harding, florian mohnert, dieuwke hupkes, and willem zuidema. 2018. under the hood: using diagnostic classifiers to investigate and improve how language models track agreement information. in proceedings of the 2018 emnlp workshop blackboxnlp: analyzing and interpreting neural networks for nlp, pages 240-248, brussels, belgium. association for computational linguistics. goran glavaš and ivan vulić. 2021. is supervised syntactic parsing beneficial for language understanding tasks? an empirical investigation. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 3090-3104, online. association for computational linguistics. yoav goldberg. 2019. assessing bert's syntactic abilities. corr, page 4. joseph greenberg. 1963. some universals of grammar with particular reference to the order of meaningful elements. in j. greenberg, ed., universals of language. 73-113. cambridge, ma. kristina gulordava, piotr bojanowski, edouard grave, tal linzen, and marco baroni. 2018a. colorless green recurrent networks dream hierarchically. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1195-1205, new orleans, louisiana. association for computational linguistics. kristina gulordava, piotr bojanowski, edouard grave, tal linzen, and marco baroni. 2018b. colorless green recurrent networks dream hierarchically. arxiv:1803.11138 [cs]. ashim gupta, giorgi kvernadze, and vivek srikumar. 2021. bert & family eat word salad: experiments with text understanding. proceedings of the aaai conference on artificial intelligence, 35(14):12946-12954. suchin gururangan, swabha swayamdipta, omer levy, roy schwartz, samuel bowman, and noah a. smith. 2018. annotation artifacts in natural language inference data. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), pages 107-112, new orleans, louisiana. association for computational linguistics. isabel papadimitriou and dan jurafsky. 2020. learning music helps you read: using transfer to study linguistic structure in language models. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 6829-6839, online. association for computational linguistics. prasanna parthasarathi, koustuv sinha, joelle pineau, and adina williams. 2021. sometimes we want ungrammatical translations. in findings of the association for computational linguistics: empirical methods in natural language processing (emnlp). ethan perez, douwe kiela, and kyunghyun cho. 2021. rissanen data analysis: examining dataset characteristics via description length. in proceedings of the thirty-eighth international conference on machine learning (icml). matthew peters, mark neumann, mohit iyyer, matt gardner, christopher clark, kenton lee, and luke zettlemoyer. 2018. deep contextualized word representations. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 2227-2237, new orleans, louisiana. association for computational linguistics. thang m. pham, trung bui, long mai, and anh nguyen. 2020. out of order: how important is the sequential order of words in a sentence in natural language understanding tasks? arxiv:2012.15180 [cs]. tiago pimentel, naomi saphra, adina williams, and ryan cotterell. 2020a. pareto probing: trading off accuracy for complexity. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 3138-3153, online. association for computational linguistics. tiago pimentel, josef valvoda, rowan hall maudslay, ran zmigrod, adina williams, and ryan cotterell. 2020b. information-theoretic probing for linguistic structure. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4609-4622, online. association for computational linguistics. adam poliak, jason naradowsky, aparajita haldar, rachel rudinger, and benjamin van durme. 2018. hypothesis only baselines in natural language inference. in proceedings of the seventh joint conference on lexical and computational semantics, pages 180-191, new orleans, louisiana. association for computational linguistics. pranav rajpurkar, jian zhang, konstantin lopyrev, and percy liang. 2016. squad: 100,000+ questions for machine comprehension of text. in proceedings of the 2016 conference on empirical methods in natural language processing, pages 2383-2392, austin, texas. association for computational linguistics. jorma rissanen. 1984. universal coding, information, prediction, and estimation. ieee transactions on information theory, 30(4):629-636. anna rogers, olga kovaleva, and anna rumshisky. 2020. a primer in bertology: what we know about how bert works. transactions of the association for computational linguistics, 8:842-866. 1298-1308, minneapolis, minnesota. association for computational linguistics. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.