  We study how masking and predicting tokens in an unsupervised fashion can
give rise to linguistic structures and downstream performance gains. Recent
theories have suggested that pretrained language models acquire useful
inductive biases through masks that implicitly act as cloze reductions for
downstream tasks. While appealing, we show that the success of the random
masking strategy used in practice cannot be explained by such cloze-like masks
alone. We construct cloze-like masks using task-specific lexicons for three
different classification datasets and show that the majority of pretrained
performance gains come from generic masks that are not associated with the
lexicon. To explain the empirical success of these generic masks, we
demonstrate a correspondence between the Masked Language Model (MLM) objective
and existing methods for learning statistical dependencies in graphical models.
Using this, we derive a method for extracting these learned statistical
dependencies in MLMs and show that these dependencies encode useful inductive
biases in the form of syntactic structures. In an unsupervised parsing
evaluation, simply forming a minimum spanning tree on the implied statistical
dependence structure outperforms a classic method for unsupervised parsing
(58.74 vs. 55.91 UUAS).
