Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models (2022)

Abstract: We investigate what kind of structural knowledge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural properties that mimic natural language, pretrain encoders on the data, and see how much performance the encoder exhibits on downstream tasks in natural language.Our experimental results show that pretraining with an artificial language with a nesting dependency structure provides some knowledge transferable to natural language.A follow-up probing analysis indicates that its success in the transfer is related to the amount of encoded contextual information and what is transferred is the knowledge of position-aware context dependence of language.Our results provide insights into how neural network encoders process human languages and the source of cross-lingual transferability of recent multilingual language models.

## introduction pretrained language models @xcite @xcite have demonstrated strong empirical performance not only within a language but also across languages. language models pretrained with a mix of monolingual corpora, such as multilingual bert, exhibit a decent zero-shot cross-lingual transfer capability, i.e., a model fine-tuned in a single source language (l1) can solve the task in another language (l2) @xcite . surprisingly, the transfer happens without lexical overlaps between l1 and l2 @xcite or even without joint pretraining @xcite : an encoder only pretrained on l1 can be transferred to l2 without any parameter updates. these results suggest that, whether the encoder is trained on single or multiple languages, it learns some transferable knowledge about language. figure 1 : transfer from artificial language to natural language. the artificial language encodes some structural properties (e.g., token distributions, dependency structures) and we study how the learning of such properties can be transferred to natural language. however, the characteristics of such transferable knowledge are still underexplored. recent studies with the probing methodology @xcite have revealed that multilingual bert captures languageindependent linguistic structures such as universal dependency relations @xcite and subjecthood @xcite , but it remains unknown whether learning such linguistic properties actually contributes to the performance, and whether there exists more abstract knowledge transferred across languages. in this study, we try to shed light on these questions with the framework of the test for inductive bias via language model transfer @xcite , focusing on designing artificial languages with natural-language-like structural properties (figure 1 ). we pretrain encoders with artificial languages and transfer the encoders to natural language tasks with their parameters frozen. this enables us to see how learning the specific structural properties of the artificial language affects the downstream performance. specifically, we explore whether it is beneficial for the encoder to know the following two characteristics of natural language: word distributions and latent dependency structures. we design artificial languages that represent such characteristics and perform an extensive study with different encoder architectures (lstm and transformer) pretraining objectives (causal and masked language modelings). the contribution is summarized as follows: ## causal language model pretraining with artificial language in this section, we complement the study of papadimitriou and jurafsky (2020). while they studied the inductive bias learned in lstm encoders with some artificial languages, here we provide additional studies with the newly introduced loglinear and dependency artificial languages, and the transformer encoder. ## masked language model pretraining with artificial language we proceed to investigate transfer learning from artificial languages in one of the most successful pretraining paradigms, masked language modeling (mlm) @xcite to see if we can observe similar trends to what we see in the clm experiment ( §4). ## discussion and future work in this paper, we studied what kind of structural properties in pretraining data is useful to train encoders for natural language tasks. we have found that to achieve decent results, l1 needs at least statistical dependency in a sentence ( §4), and having the head-to-tail dependency with the nesting structure is further beneficial ( §4 and §5). the probing experiment in §6 suggests that the encoders trained with languages with the above characteristics are good at capturing the positions and identities of the context words. from these observations, we suggest a tentative answer to the initial research question: what knowledge in pretrained encoders are transferred across different languages? that is position-aware context dependence of language, in other words, "tokens in a sequence can be characterized by its neighbor tokens at specific positions". we think that it can explain the success of transferring the encoder across languages to some extent. to solve natural language tasks, it is often useful to characterize words in a sentence by the words around them. for example, to understand the semantics of a sentence, it would be useful to look for the subject by looking for a noun that precedes the word is; to parse a sentence, a word can be identified as a noun because it follows the article the. if the encoder computes the output representation of a word in a sentence by aggregating the information from its surrounding words, that should be a useful inductive bias to solve most nlp tasks in any language. also, it is easy to imagine that the knowledge of position-aware context dependence gives a reasonable prior for solving sequence modeling problems in other domains, which may explain the success of cross-modality transfer of language models @xcite . of course, we do not expect that the knowledge of position-aware context dependence explains every aspect of the success of cross-lingual transfer. as future work, we need further investigation for a more fine-grained view of the transferred knowledge. important questions include how much the model size affects the transferability of the encoder or if there is any difference in the knowledge transferred among different downstream tasks.