On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies (2021)

Abstract: We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained performance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the Masked Language Model (MLM) objective and existing methods for learning statistical dependencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful inductive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the implied statistical dependence structure outperforms a classic method for unsupervised parsing (58.74 vs. 55.91 UUAS).

## introduction pretrained masked language models @xcite have benefitted a wide range of natural language processing (nlp) tasks @xcite @xcite . despite recent progress in understanding what useful information is captured by mlms @xcite , it remains a mystery why task-agnostic masking of words can capture linguistic structures and transfer to downstream tasks. one popular justification of mlms relies on viewing masking as a form of cloze reduction. ## related works theories inspired by cloze reductions. cloze reductions are fill-in-the-blank tests that reformulate an nlp task into an lm problem. existing work demonstrates that such reductions can be highly effective for zero/few-shot prediction @xcite as well as relation extraction @xcite . these fill-in-the-blank tasks provide a clear way by which lms can obtain supervision about downstream tasks, and recent work demonstrates how such implicit supervision can lead to useful representations @xcite . more general arguments by @xcite show these theories hold across a range of self-supervised settings. while these theories provide compelling arguments for the value of pre-training with cloze tasks, they do not provide a clear reason why uniformly random masks such as those used in bert provide such strong gains. in our work, we quantify this gap using lexicon-based cloze-like masks and show that cloze-like masks alone are unlikely to account for the complete success of mlm since generic and non-cloze masks are responsible for a substantial part of the empirical performance of mlms. while existing theories for word embeddings cannot be applied to mlms, we draw inspiration from them and derive an analogous set of results. our work shows a correspondence between mlm objectives and graphical model learning through conditional mutual information, as well as evidence that the conditional independence structure learned by mlms is closely related to syntactic structure. ## analysis in the previous section, we saw that cloze-like masks do not necessarily explain the empirical success of mlms with uniform masking strategies. understanding uniform masking seems challenging at first, as uniform-mask mlms seem to lack task-specific supervision and is distinct from existing unsupervised learning methods such as word embeddings (which rely upon linear dimensionality reduction) and autoencoders (which rely upon denoising). however, we show in this section that there is a correspondence between mlm objectives and classic methods for graphical model structure learning. as a consequence, we demonstrate that mlms are implicitly trained to recover statistical dependencies among observed tokens. ## experiment we now test two predictions from our analyses. first, similar to our observation in the case study, we show that cloze-like masks do not explain the success of uniform masks on three real-world datasets. second, our alternative view of relating mlm to graphical models suggests that statistical dependencies learned by mlms may capture linguistic structures useful for downstream tasks. we demonstrate this by showing that mlms' statistical dependencies reflect syntactic dependencies. ## discussion and conclusion we study how mlm with uniform masking can learn useful linguistic structures and inductive biases for downstream tasks. our work demonstrates that a substantial part of the performance gains of mlm pretraining cannot be attributed to taskspecific, cloze-like masks. instead, learning with task-agnostic, generic masks encourages the model to capture direct statistical dependencies among tokens, and we show through unsupervised parsing evaluations that this has a close correspondence to syntactic structures.