The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining (2023)

Abstract: We analyze the masked language modeling pretraining objective function from the perspective of the Distributional Hypothesis. We investigate whether the better sample efficiency and the better generalization capability of models pretrained with masked language modeling can be attributed to the semantic similarity encoded in the pretraining data’s distributional property. Via a synthetic dataset, our analysis suggests that distributional property indeed leads to the better sample efficiency of pretrained masked language models, but does not fully explain the generalization capability. We also conduct an analysis over two real-world datasets and demonstrate that the distributional property does not explain the generalization ability of pretrained natural language models either. Our results illustrate our limited understanding of model pretraining and provide future research directions.

## introduction despite the rise of the prompting paradigm with the scaling breakthrough of very large language models @xcite , understanding the mechanism of model fine-tuning remains to be an important endeavor. fine-tuning relatively small models such as bert @xcite @xcite has significant practical implications since these models are computationally more efficient than large language models such as gpt-3 @xcite . distilling a large language model to a small model by fine-tuning is also a promising direction @xcite . the development of large language models also involves fine-tuning, e.g. doing reinforcement learning from human feedback @xcite . in this work, we seek to understand how pretraining benefits downstream fine-tuning. previous analyses of pretrained model focus on probing the pretrained representations @xcite @xcite @xcite @xcite . however, these models are rarely used as a static feature extractor. practitioners often fine-tune them with downstream tasks. we analyze whether mask language model pretraining allows models to leverage semantic knowledge derived from the distribution of words in the pretraining data for downstream tasks. this research question follows the distributional hypothesis @xcite . it postulates that semantically related words have a similar context distribution, which we elaborate more rigorously as a distributional property of pretraining data in §2. because it draws connections between word semantics and data distribution, it has been widely accepted as an explanation for the efficacy of non-contextualized (type-based) word embeddings @xcite such as word2vec @xcite and glove @xcite . recently, @xcite shows that word order does not greatly affect the efficacy of masked language model pretraining @xcite , thus conjecturing that the distributional hypothesis could be an explanation. in this work, we continue studying this conjecture by investigating whether pretrained models can utilize the semantic knowledge derived from the data distribution as suggested by the distribution hypothesis. among different types of semantic knowledge the distribution of the pretraining data may encode, we focus on semantic equivalence, i.e., synonym, which @xcite suggests to be encoded in the data distribution. in §3, we theoretically show that prior knowledge about semantic equivalence alone is enough to lead to two desirable properties that pretrained language models have: better sample efficiency and generalization capability @xcite @xcite . in §4, as our first step to understanding how masked language modeling training objective exploits the distributional property of the pretraining data, we set up a toy experiment. we construct a synthetic pretraining corpus and a downstream task. in §5, we show that the distributional property explains the sample efficiency of pretrained masked language models, but it does not always lead to better generalization on its own; it still depends on the type of distribution shifts. our results also suggest that better parameter initialization statistics @xcite do not explain the benefit of masked language model pretraining either. in §6, we conduct a similar experiment in the real-world setting. we analyze a pretrained bert model and two real-world tasks sst-2 @xcite and mnli @xcite . we find that the semantic (in)equivalence the model learns from the pretraining task is independent of how a fine-tuned model treats words as synonyms. this indicates that pretrained models do not generalize better by modeling the distributional property of the pretraining data either. to summarize, our main findings include: i) we show that semantic equivalence encoded in the distributional property of the pretraining data makes pretrained models more sample-efficient. ii) there is a type of generalization capability independent of the distributional property of the pretraining data. ## properties of pretraining data and downstream tasks since we focus on semantic equivalence in language, we use the concept of "synsets" from word-net @xcite , i.e., the sets of synonyms. here we extend the concept of synsets to allow each synset to contain multi-token elements, such as phrases or sentences. elements in a synset are features and each of them is a sequence of tokens (or a single token). having these semantic equivalence sets has the following implications: in the pretraining data: the distributional hypothesis suggests that the pretraining data has a distributional property: for any two features a, b in the same synset and any @xmath0, the distribution of their neighboring words satisfies for example, the phrase "is delicious" has the same meaning as the phrase "taste good". therefore, if the training data has this distributional property, then we expect its distribution to satisfy p(x|"is delicious") ≈ p(x|"tastes good"). in a downstream task: given an input x for a downstream task, substituting a feature a with another feature b in the same synset does not change the meaning of x because a and b are synonyms. therefore, the substitution does not change the label of x either: ## the benefit of modeling semantic equivalence in this section, we show why having prior knowledge about the semantic equivalence sets (synsets) helps downstream performance. ## analyzing models for real-world datasets in § 5, we show that the distributional property in eq. 1 can improve sample efficiency, at least when the feature is at the single-token level. we posit that this conclusion applies to the real-world scenario where the data also has this property. however, the negative results regarding the generalization capability may be due to the oversimplification of our toy setting. therefore, in this section, we study generalization in real-world scenarios. unfortunately, it is difficult to conduct a fully controlled experiment as we do with the toy settings. it is impossible to find out all the synsets in a natural language. it is also difficult to divide features into isomorphic subsets as the feature sets φ a and φ b we have in the toy settings. therefore, we will measure the relationship between the distributional property in eq. 1 and generalization in an indirect manner. ## related work this work aims to inspect how mlm pretraining infuses inductive bias for downstream tasks. it is different from previous studies that treat pretrained models as static models. for example, @xcite and @xcite study the efficacy of mlm theoretically by analyzing the distribution predicted by the model. tenney et al. (2019b,a); liu et al. (2019a); hewitt and manning (2019); wu et al. (2020); zhang et al. (2021) study the rich linguistic structure encoded in the representation. some works focus on the fine-tuning process but the pretraining loss. for instance, hao et al. (2019) shows mlms have better loss landscape. aghajanyan et al. (2021) shows that pretrained models have lower intrinsic dimensionality. malladi et al. @xcite shows neural tangent kernels @xcite can explain mlm's efficacy to some extent. some works study the connection between pretraining and downstream tasks. @xcite show that non-language pretraining tasks help nlp downstream tasks, while lu et al. show that language models help non-language downstream tasks. @xcite uses some linguistic tasks to show that the linguistic structures in mlms can serve as an inductive bias. @xcite investigate whether mlms' efficacy can be explained by the task-relevant cloze-like masks alone. our work follows this line and explains mlms' general machine-learning characteristics. beyond the scope of nlp, the concept of connecting training examples is very common in machine learning. for example, co-training @xcite uses multiple views of features to build connections between training samples. recently, @xcite show that contrastive self-supervised learning reduces sample efficiency by pulling the representation of similar samples together. similarly, @xcite shows that contrastive learning connects samples in different domains and thus helps generalization. however, those studies mainly focus on contrastive learning for continuous data instead of discrete languages. ## discussion we showed in a synthetic experiment that pretraining with a masked language model objective allows the model to make use of semantic equivalence that simple distributions (e.g. markov models) can encode to improve models' sample efficiency. however, we found that it can not explain generalization in the presence of distribution shifts. our work leads to two future research directions. first, we showed the limitations of our understanding of pretraining. what semantic relationships are important, how data distributions encode those relationships, and how modeling data distributions help downstream performance remain open questions. second, our results in §6 show the unpredictability of models' generalization behavior. investigating this behavior may help us develop more robust nlp models.