L earning M usic H elps Y ou R ead: U sing Transfer to Study Linguistic Structure in Language Models (2020)

Abstract: We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models. We train LSTMs on non-linguistic data and evaluate their performance on natural language to assess which kinds of data induce generalizable structural features that LSTMs can use for natural language. We find that training on non-linguistic data with latent structure (MIDI music or Java code) improves test performance on natural language, despite no overlap in surface form or vocabulary. To pinpoint the kinds of abstract structure that models may be encoding to lead to this improvement, we run similar experiments with two artificial parentheses languages: one which has a hierarchical recursive structure, and a control which has paired tokens but no recursion. Surprisingly, training a model on either of these artificial languages leads the same substantial gains when testing on natural language. Further experiments on transfer between natural languages controlling for vocabulary overlap show that zero-shot performance on a test language is highly correlated with typological syntactic similarity to the training language, suggesting that representations induced by pre-training correspond to the cross-linguistic syntactic properties. Our results provide insights into the ways that neural models represent abstract syntactic structure, and also about the kind of structural inductive biases which allow for natural language acquisition.

## introduction understanding how neural language models learn and represent syntactic structure is an important analytic question for nlp. recent work has directly probed the internal activations of models (conneau 1 we release code to construct the corpora and run our experiments at https://github.com/toizzy/ tilt-transfer ppl when tested on spanish (l2) (lower is better) figure 1 : we find that lstm lms can utilize various types of non-linguistic structure to help learn to model human language, and that nested hierarchical structure does not lead to more expressive encodings than flat, head-dependency pair structure. we also find that lstm lms learn representations that correlate with typological syntactic feature distance, allowing them to transfer more effectively from languages which are grammatically similar. @xcite @xcite , or fed them curated inputs that depend on complex syntax @xcite @xcite , in order to uncover latent syntactic awareness. we propose a different approach: we measure the structural awareness of a language model by studying how much this structure acts as an inductive bias to improve learning when we transfer from one language or symbolic system to another. we train lstm models on data with varying degrees of language-like structure (music, java code, nested symbols), and then evaluate their performance on natural language. before evaluation, we freeze the lstm parameters and fine-tune the word embeddings on the evaluation language. this lets us see if the training data induces language-like lstm lin lin lstm lin lin lstm lin lin lstm lin lin lstm lin lin lstm lin lin lstm lstm lstm ① train on l1 ③ finetune word embeddings on l2 spanish spanish spanish spanish test music parentheses japanese ② freeze lstm parameters ④ test on l2 figure 2: diagram illustrating our training procedure: k models are trained on k l1 languages, and then their lstm weights are frozen while their linear layers are finetuned on a common l2 language (in our case, we always use spanish as the l2). we can then compare their performance on the common l2. lastly, in transfer experiments between different human languages, we find that transfer is better between languages that are syntactically typologically similar, even with no vocabulary overlap. this suggests that models have the ability to form representations of typologically sensible properties rather than relying on ad-hoc or non-natural representations. for this result we draw on recent interlingual work such as @xcite , and @xcite , extending it to use typological distance to turn these observations into quantitative probes. the tilt method allows us to ask a complementary set of questions to those answered by current analysis methods. tilts demonstrate the abstract structural notions that lstms can learn, rather than probing for the manifestation of a particular known structure, as in most current methods. by examining the pretraining structures that give lstms a better ability to model language, we also contribute to the more general cognitive question of what structural inductive biases a learner needs to be able to easily acquire human language. ## architecture and training our methodology consists of training lstm language models on k different first languages (l1s) which include natural languages, artificial languages, and non-linguistic symbol systems, and testing the performance of these models on a common second (l2) language. in our case, we used spanish as the common l2. before testing on the l2 test set, we fine-tune the linear embedding layer of the models on the l2 training set, while keeping the lstm weights frozen. this aligns the vocabulary of each model to the new language, but does not let it learn any structural information about the l2 language. though word embeddings do contain some grammatical information like part of speech, they do not contain information about how to connect tokens to each other -that information is only captured in the lstm. figure 2 illustrates our training process. 3 we vary the l1 languages and maintain a common l2 (instead of the other way around) in order to have a common basis for comparison: all of the models are tested on the same l2 test set, and therefore we can compare the perplexity scores. we run @xmath0. any high-resource human language would have provided a good common l2, and spanish works well for our human languages experiments due to the fact that many higher-resource languages fall on a smooth gradation of typological distance from it (see we use the awd-lm model @xcite ) with the default parameters of 3 lstm layers, 300dimensional word embeddings, a hidden size of 1,150 per layer, dropout of 0.65 for the word embedding matrices and dropout of 0.3 for the lstm parameters. we used sgd and trained to convergence, starting the learning rate at the default of 30 and reducing it at loss plateau 5 times. much of the work on multilingual transfer learning has speculated that successes in the field may be due to vocabulary overlap (see for example @xcite ). since our work focuses mostly on syntax, we wanted to remove this possibility. as such, we shuffle each word-to-index mapping to use disjoint vocabularies for all languages: the english word "chile" and the spanish word "chile" would map to different integers. this addresses the confound of vocabulary overlap, as all language pairs have zero words in common from the point of view of the model. since the vocabularies are totally separated between languages, we align the vocabularies for all l1-l2 pairs by finetuning the word embeddings of all the pretrained models on the spanish (l2) training data, keeping the lstm weights frozen. by doing this, we remove the confound that would arise should one language's vocabulary randomly happen to be more aligned with spanish than another's. these controls ensure that lexical features, whether they be shared vocabulary or alignment of randomly aligned indices, do not interfere with the experimental results which are meant to compare higher-level syntactic awareness. ## experiment 1: random baselines we run our method on a random baseline l1: a corpus where words are sampled uniformly at random. this gives us a baseline for how much information we gain finetuning the word embeddings to the l2, when there has not been any structurally biasing input to the lstm from the l1. we also examine the importance of vocabulary distribution by training on a random corpus that is sampled from a zipfian distribution. human languages are surprisingly consistent in sharing a roughly zipfian vocabulary distribution, and we test how pretraining on this distribution affects the ability to model human language. foot_2 ## experiment 2: non-linguistic structure in this experiment, we test the performance of lstms on spanish when they have been trained on music and on code data. while music data especially is very different from human language on the surface level, we know that music and code both contain syntactic elements that are similar to human language. foot_3 by comparing performance to our random baselines, we ask: can lstms encode the abstract structural features that these corpora share with natural language in a generalizable way that's usable to model human language? ## experiment 3: recursive structure in this experiment, we isolate and assess possible structural features of music and code that may explain the results of experiment 2. the most widelyknown structural hypothesis is the claim of @xcite that the narrow language faculty in humans (the inductive bias in the mind/brain that allows humans to acquire and develop language) can be reduced to just recursion. given the prominence of such theories, it is natural to ask: is it the matrix of 50,000 rows, but during training only ever sees words 1-310, meaning that much of the word embedding space has never been seen by the lstm part of the model. to test this possibility, we create a simple recursive corpus: a nesting parentheses corpus of hierarchically nesting matching symbols, and run the same experimental setup as we did for experiments 1 and 2 foot_6 . we find that plain recursion, even when the corpus has no other structural subtleties, is indeed a sufficient condition for inducing the kinds of structural transfer we observed in experiment 2. recursion is a sufficient quality, but is it the only explanation for our results? we also create a control corpus: a flat parentheses corpus, which has similar pairs of matching parentheses, but which do not nest hierarchically and projectively (the difference between the two corpora is visually illustrated in figure 3 ). we surprisingly find that this non-recursive corpus induces the same amount of structural transfer as the recursive nesting parentheses, which emphasizes the importance of pairing, head-dependency type structure in the linguistic structural embeddings of lstms. ## experiment 4: human languages to further analyze what kinds of generalizable structure lstms can infer, we run experiments in transferring zero-shot between human languages. we ask: can lstms infer and use fine-grained syntactic similarities between typologically similar languages? previous work @xcite indicates that transfer is more successful between related languages. we control for vocabulary overlap, and use typological syntactic difference as a quantitative probe to ask: are fine-grained syntactic similarities encoded in generalizable, transferrable ways? to answer this question, we investigate the extent to which fine-grained differences in syntactic structure cause different zero-shot transfer results. ## discussion in this work we propose the test for inductive bias via language model transfer (tilt), a novel analytic method for neural language models which tests the ability of a model to generalize and use structural knowledge. we pretrain lstms on structured data, and then use the frozen lstm weights to model human language. in doing so, we treat the frozen lstm weights as the only structural faculty available to a human language model, and assess if the induced structure is general enough to be used to model human language. our experiments are cross-lingual and crossmodal in nature, not searching for representations of high-level features in one language, but for representations that encode general ideas of structure. while the majority of past work analyzing the structural abilities of neural models looks at a model's treatment of structural features that are realized in specific input sentences, our method compares the encoding and transfer of general grammatical features of different languages. by using tilts, we do not have to identify a structural feature of interest and investigate if it is being encoded, but instead asses if generalizable abstract structures are encoded in one language by examining if they can be used to model human language. our work thus avoids known issues that have been pointed out with analytic methods like probing @xcite @xcite . we run experiments on natural languages, artificial languages, and non-linguistic corpora. our non-linguistic and artificial language experiments suggest three facets of the structural encoding ability of lstm lms. first, that vocabulary distribution has a very minor effect for modelling human language compared to structural similarity. second, that models can encode useful language modelling information from the latent structure inherent in non-linguistic structured data, even if the surface forms are vastly differing. last, that encodings derived from hierarchically structured tokens are equally useful for modelling human language as those derived from texts made up of pairs of tokens that are linked but non-hierarchical. running experiments on a range of human languages, we conclude that the internal linguistic representation of lstm lms allows them to take advantage of structural similarities between languages even when unaided by lexical overlap. our results on the parentheses corpora do not necessarily provide proof that the lstms trained on the nesting parentheses corpus aren't encoding and utilizing hierarchical structure. in fact, previous research shows that lstms are able to suc-cessfully model stack-based hierarchical languages @xcite @xcite . what our results do indicate is that, in order for lstms to model human language, being able to model hierarchical structure is similar in utility to having access to a non-hierarchical ability to "look back" at one relevant dependency. these results shine light on the importance of considering other types of structural awareness that may be used by neural natural language models, even if those same models also demonstrate the ability to model pure hierarchical structure. our method could be used to test many other hypotheses regarding neural language models, by choosing a discerning set of pretraining languages. a first step in future work would be to test if the results of this paper hold on transformer architectures, or if instead transformers result in different patterns of structural encoding transfer. future work expanding on our results could focus on ablating specific structural features by creating hypothetical languages that differ in single grammatical features from the l2, in the style of galactic dependencies @xcite , and testing the effect of structured data that's completely unrelated to language, such as images. our results also contribute to the long-running nature-nurture debate in language acquisition: whether the success of neural models implies that unbiased learners can learn natural languages with enough data, or whether human abilities to acquire language given sparse stimulus implies a strong innate human learning bias @xcite . the results of our parentheses experiments suggest that simple structural head-dependent bias, which need not be hierarchical, goes a long way toward making language acquisition possible for neural networks, highlighting the possibility of a less central role for recursion in language learning for both humans and machines.