Examining the Inductive Bias of Neural Language Models with Artificial Languages (2021)

Abstract: Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages. Investigation of these biases has proved complicated due to the many variables that appear in the experimental setup. Languages vary in many typological dimensions, and it is difficult to single out one or two to investigate without the others acting as confounders. We propose a novel method for investigating the inductive biases of language models using artificial languages. These languages are constructed to allow us to create parallel corpora across languages that differ only in the typological feature being investigated, such as word order. We then use them to train and test language models. This constitutes a fully controlled causal framework, and demonstrates how grammar engineering can serve as a useful tool for analyzing neural models. Using this method, we find that commonly used neural architectures exhibit different inductive biases: LSTMs display little preference with respect to word ordering, while transformers display a clear preference for some orderings over others. Further, we find that neither the inductive bias of the LSTM nor that of the transformer appear to reflect any tendencies that we see in attested natural languages.

## introduction modern neural architectures used for language modeling, e.g. transformer-based language models @xcite and language models based on long-short term memory (lstm) @xcite , are intrinsically black boxes. this makes it difficult to understand whether their structure leads to an inductive bias which results in certain types of language being easier to learn and model. to make this point more plainly, we cannot easily conclude much about whether an lstm language model will perform better on svo or sov languages by simply examining its structure. moreover, satisfactorily investigating the inductive bias of neural models has the potential to yield useful insight into how they work. in this work, we explore whether neural language models exhibit biases towards certain types of languages in a novel causal framework through the use of artificial languages. one of the key problems involved in investigating the effect of typological features on language model performance is the difficulty in isolating only the features being investigated, without influence from other features of the languages being investigated or the data being used. for example, if one were to compare language model performance on english, an svo language, and japanese, an sov language, it would be difficult to directly attribute differences in performance to the difference in word ordering alone. this is because english and japanese also differ in many other typological dimensions, such as how subjects are marked, the extent of subject-verb agreement and use of postpositions or prepositions, which could contribute to the difference in performance. indeed, recent correlational studies have failed to find an effect between language model performance and typological features @xcite . moreover, the sentences used for training and testing may differ in content, style or information density, which could further contribute to differences in performance. thus, we offer a study investigating the inductive biases of language models through the construction of artificial languages. our approach involves creating small context-free grammars resembling subsets of attested languages, which we then use to train and evaluate language models. in an approach inspired by @xcite framework of principles and parameters, we imbue our grammars with "switches" that indicate how to permute the ordering of the non-terminals in a given production. through generating grammars with all possible combinations of these switches, we can create artificial languages of differing typological profiles. this experimental paradigm allows us to conduct carefully controlled studies by varying only the typological parameter and make a causal claim. using our method, we investigate inductive biases related to the head-directionality of several constructions. we find that lstm-based architectures show little bias towards any particular ordering, achieving similar average perplexities on all grammar variations tested. this contradicts recent findings by @xcite who find lstms have a preference for svo word order. conversely, we find that performance of transformer-based architectures varies significantly across our artificial languages; this is visualized in figure 1 . this indicates that some combinations of the switches result in languages with word orders that are harder for the transformer to model than others. our analysis suggests that neither the performance of the transformer-based architectures nor of the lstm-based architectures reflects any known tendencies in attested natural languages, with the best performance being achieved on languages with the rarely-attested ovs sentence ordering. importantly, our method exposes that transformer-based language models and lstm-based language models have vastly different inductive biases, a result that has not been clearly stated in the nlp literature. ## discussion differences between architectures. it is clear from figure 3 that the transformer-and lstmbased models do not show the same inductive biases with respect to the switches we investigated. across all possible configurations of the switches, lstms achieve very similar average perplexities, suggesting that they have little preference for any particular set of constituent orderings. in contrast, the average perplexities achieved by the transformers vary considerably between grammars. this demonstrates clearly that the two models exhibit distinctly different preferences with regard to orderings of words within in a sentence. further, the clear contrast between the coefficients obtained by the mixed-effects models for transformers and lstms (shown in figure 4a and figure 4b , respectively) demonstrates a stark contrast between the two models. none of the switches investigated, or their first-order interactions, appear to have a substantial effect on the scores obtained in the case of the lstm-based models, whereas the transformer-based models are clearly affected to a much greater degree by the configuration of these switches. given that these two architectures are both commonly used for similar tasks, such a difference in their inductive biases is noteworthy. ## conclusion we propose a novel methodology for the investigation of the inductive bias of language models using the technique of creating carefully controlled artificial languages. this approach allows for the elimination of differences in corpora between languages and means that typological variation between languages can be restricted exclusively to the typological features being investigated. we use this methodology to investigate the inductive bias of two neu-ral architectures which are commonly used for this task: lstms and transformers. we found that these two models have starkly different inductive biases with respect to word order, with the lstm showing little variation in performance across word order, while the performance of the transformer varied significantly across artificial languages.