  A possible explanation for the impressive performance of masked language
model (MLM) pre-training is that such models have learned to represent the
syntactic structures prevalent in classical NLP pipelines. In this paper, we
propose a different explanation: MLMs succeed on downstream tasks almost
entirely due to their ability to model higher-order word co-occurrence
statistics. To demonstrate this, we pre-train MLMs on sentences with randomly
shuffled word order, and show that these models still achieve high accuracy
after fine-tuning on many downstream tasks -- including on tasks specifically
designed to be challenging for models that ignore word order. Our models
perform surprisingly well according to some parametric syntactic probes,
indicating possible deficiencies in how we test representations for syntactic
information. Overall, our results show that purely distributional information
largely explains the success of pre-training, and underscore the importance of
curating challenging evaluation datasets that require deeper linguistic
knowledge.
