Injecting structural hints: Using language models to study inductive biases in language learning (2023)

Abstract: Both humans and transformer language models are able to learn language without explicit structural supervision. What cognitive inductive biases make this learning possible? Here, we examine the effect of different inductive learning biases by actively controlling the inductive biases of artificial learners: we structurally bias models by pretraining on synthetic formally-structured data, and evaluate these structural biases by fine-tuning on three typologically-distant human languages: English, Japanese, and Basque. We investigate the effect on downstream language perplexity of three types of inductive bias: 1) recursive, hierarchical processing 2) unrestricted token-token dependencies that can’t be modeled by context-free grammars, and 3) a Zipfian power-law vocabulary distribution. We show that complex, non-context-free interactions between tokens form the best inductive biases. Our study leverages the capabilities of transformer models to run controlled language learning experiments that are not possible to run on humans, and surfaces hypotheses about the structures that facilitate language learning in both humans and machines.

## introduction natural languages are complex and structured systems which humans learn without direct structural supervision. it is a fundamental and open problem in linguistics and cognitive science to understand the inductive biases that make such learning possible: what structural predispositions does a successful language learner need to start with? in this work, we shed light on this cognitive problem using artificial learners. using our method of structural injection, we causally intervene in transformer language models and manipulate their structural inductive biases, before training on natural language. we predispose transformers with three structural biases from the cognitive literature: a bias for recursive processing, a bias for keeping track untrained gpt-2 model formal language structurallybiased learner wikitext structural principle: recursion, or context-sensitivity, or ... pretraining: inject structural inductive bias metric: perplexity after fine-tuning figure 1 : our method: we take a gpt-2-sized model and pretrain it with a formal language corpus (see @xcite 3 for examples). we then take these pretrained models and fine-tune them on wikipedia data to asses each formal structure as an inductive bias for learning english, japanese, and basque. the lawyer that the man that the dog bit hired was disbarred in experiment 1, we disentangle the effects of recursion and context-sensitivity: both occur in language, but which one is more useful as a sole learning bias? language is characterized by recursive structures in context-free constituent relationships like those in figure 2a , and some linguistic theories posit that recursive processing is a crucial (and perhaps the only) inductive bias that makes human language learning possible @xcite . however, it is widely hypothesized that human language is mildly context sensitive: while there is recursive structure, there are also noncontext-free relationships between tokens, both in syntactic structure (figure 2b ) and in the structure of meaning and discourse relationships (figure 2c ) @xcite @xcite @xcite . we compare the two principles of recursion and non-context-free relationships, and find that an inductive bias for non-context-free crossing dependencies is better for downstream language learning. however, both inductive biases greatly outperform random and regular language controls. in experiment 2, we combine recursion and context-sensitivity and ask: is a recursive inductive bias better with slight context-sensitivity? we show that there is a significant improvement in downstream language learning if we add just 1% of a bias for crossing dependencies, breaking the constituent structure of the other 99% recursive bias we give the learner. even when learners are mostly biased towards recursion, they learn language faster when their bias includes constituentbreaking context-sensitive structures. in experiment 3, we test the effect of inductive biases in vocabulary distribution: does a bias towards a human-like zipfian vocabulary distribution @xcite help language learning? lan-guage is structured not only in how tokens relate, but also in the structure of the distribution that tokens are drawn from, a cognitive bias that is especially significant for memory-based theories of grammar @xcite @xcite . we show that a zipfian pretraining bias makes models better at downstream language learning even when there is no correspondence between the pretraining and fine-tuning vocabularies. while our experiments work with computational models, the question that we are examining is about humans: what are the possible inductive biases that make human language learning possible without explicit structural supervision? using computational models, we can investigate this question through empirically manipulating the inductive bias of a language learner -a causal experimental route that is not possible when working with humans. results from such experiments can act as firstly as proofs of concept, showing how language learning is or isn't possible with different inductive biases. secondly, such experiments help with hypothesis generation: with structural injection, we can test arbitrary inductive biases in a theory-independent way (see @xcite @xcite @xcite , for further discussions on the role of neural nlp models in cognitive science). in section 8 we discuss how our method of leveraging artificial learners to causally investigate inductive biases adds a new direction to the rich prior literature on inductive bias in neural learners. in summary, our findings indicate that biases for complex token-token interactions, whether these involve recursion or not, form a powerful inductive bias for learning natural language. crucially, our results are compatible with and can inform a wide variety of cognitive architectures: models in which structural inductive biases in humans arise from prior statistical learning @xcite , models in which they arise from other aspects of cognition or communication @xcite @xcite , and models in which they are presumed to be innate @xcite . 1 ## background: structural inductive bias in humans in order to use our experiments as a window into hypotheses about human inductive biases, we look at three families of structural bias from the linguistics and cognitive science literature. ## method overview: pretraining as structural inductive bias we set up experiments where we control the inductive bias of a language model learner, and examine how inductive biases influence language learning. our experiments consist of two steps: 1) pretraining a gpt-2-sized model on a formal language to inject a structural bias and 2) fine-tuning on limited natural language data (drawing on our prior methods in papadimitriou and jurafsky (2020), see figure 1 for a depiction). to instill an inductive bias in an artificial language learner, we pretrain untrained transformer language model on synthetic data which exhibits a single formal structural prin-ciple. after sufficient pretraining on such a corpus, we have a transformer model which has never seen human language, but has successfully learnt to model a type of structure. we can then train this model on natural language data, taking its pretraining to be a structural inductive bias. we fine-tune each model on three different languages (english, japanese, and basque). the final performance on each language, measured in perplexity, is an indicator of how learnable the raw language data is when a learner starts with the specific inductive bias. ## pretraining languages our experimental method rests on pretraining language models on well-chosen formal languages. here, we describe the four families of formal languages that we use for pretraining, and present examples of each of the languages in fig. 3 . ## experiment 1: disentangling recursive and context-sensitive inductive biases for our first experiment, we compare the finetuning perplexity of models pretrained on the nest language and the cross language to each other and to random and regular controls. our fine-tuning results are shown in fig. 4 . from the fine-tuning perplexity that we get with different structural inductive biases, we can present two findings: any structure has a significant effect as an inductive bias all of the models with a structural pretraining language fare significantly better than the model with the random, unstructured pretraining language. this especially surprising in the case of the rep language, which has a very simple structure that is similar to rand. however, our experiments also show that the shallow structure of rep is worse than both of our languages with more complex structural organization. comparing the performance of the nest and the cross language leads us to our next finding: context sensitivity acts as a better inductive bias than recursion our experiments disentangle the effects of recursive structure and non-context-free linking structure, and show that the non-contextfree cross language acts as a stronger inductive bias for language learning than the recursive nest language. our current methodology does not ascertain which aspects of language learning the cross language is most helpful for, which is an especially fruitful question for future work since context-sensitive linking structures arise in syntax, semantics, and discourse. understanding the role of different complex structures in influencing language learning in an in-vitro paradigm such as ours shines light on the properties of language as a learnable system under different cognitive structural assumptions. in experiment 1, we saw that the non-context-free inductive bias of the cross language is more beneficial for downstream language learning than the constituent recursion of the nest language. however, neither of these extremes encompass the more widely agreed nature of linguistic syntactic structure: likely a recursive grammar with mildly context-sensitive elements. to examine the effects of non-context-free elements in otherwise context-free grammars, we create the nest-mix-p languages. these languages follow the nest grammar, except that with probability p% an 'open' token does not follow the nest grammar and instead follows the cross grammar. this means that we sample a dependency distance and place a 'close' token for it without taking into account the constituent structure of the nest language. the nest example from figure 3a , with one cross token added and shown in green, would look like this: we test two such languages: nest-mix-1, with 1% cross tokens, and nest-mix-10, with 10% cross tokens. our results are shown in figure 5 . adding 1% cross tokens to a nest language causes it to act as a significantly better inductive bias, with an average improvement of 5 perplexity points, while adding 10% cross tokens causes an average downstream improvement of 9 perplexity points. our results show that even small amounts of context-sensitive inductive bias have a big effect on language learning, aligning with theoretical linguistics results about human language being mildly context sensitive. ## experiment 3: zipfian structural bias how does a bias towards a particular vocabulary distribution affect a language learner? we test the effect of a uniform vocabulary distribution versus a zipfian vocabulary distribution as a pretrained inductive bias. as shown in figure 6 , a zipfian pretraining distribution of random tokens predisposes a model with a better language-learning bias. crucially, there is no connection or correspondence between the pretraining and fine-tuning vocabularies: the pretraining vocabulary is made up of 500 parentheses tokens (whose frequency ranking is randomly ordered in order to make a zipfian distribution in the zipfian case), whereas the gpt-2 tokenizer's fine-tuning vocabulary is over 50k tokens. no token correspondence of specific token information is transferred between pretraining and fine-tuning as the tokens are unrelated and we re-sample the rows of the embedding matrix. since there is no correspondence, our vocabulary distribution ablations measure the effect of an abstract bias towards one type of vocabulary distribution, rather than the effect of knowing the specific distribution of tokens in a language. our experiments show that pretraining on a random zipfian corpus biases a model for better language learning than a uniform corpus, and that a structural bias for vocabulary distribution is encoded beyond the word embedding matrix and more abstractly in network parameters. due to the lack of any structure beyond vocabulary distribution in the random corpora, neither of the test cases lead to very good downstream language performance. results around combining zipfian vocabulary with other structural biases like those in experiment 1 are mixed, and we do not have clear evidence that zipfian vocabulary has a structural effect that can add on to the effect of structure in data. we discuss these results in more detail in appendix a. ## related work: inductive bias our experiments use transformer models in order to create a controllable testbed for understanding human language learning: by manually varying the inductive bias through pretraining, we can test the effect of different inductive biases in making a learner fit for language learning. our focus is different, though related, to the focus of much of the work about inductive bias in transformers. while we set the inductive bias of our learners through pretraining, the built-in structural inductive biases of untrained deep neural network architectures are unknown and difficult to assess @xcite . a rich line of past experiments work towards defining this inductive bias of neural models through empirical methods, looking at how much transformers are biased towards learning hierarchical rules @xcite @xcite or tree-structured processing @xcite , or biases towards different typological linguistic structures @xcite @xcite , and how inductive bias changes with language modeling pretraining @xcite . we build on these results by working to influence, rather than measure, the inductive bias of transformers. our experimental paradigm relies on structural transfer between pretraining and fine-tuning data in unrelated modalities, an effect demonstrated by past research. we showed in previous work how the structures of non-linguistic modalities like music and code can predispose learners for language learning (papadimitriou and jurafsky, 2020), work that has been reproduced @xcite , extended to other modalities like amino acid sequences @xcite , and between language and multiple symbolic tasks @xcite . in a similar line of inquiry, @xcite show structural transfer from the abstract task structure of summarizing random data to summarizing real natural language passages. the paradigm of structural transfer lets us study the fundamental cognitive issue of inductive bias from an exciting angle: through causal experiments where we control the inductive bias of language learners. ## discussion our experiments and methodology provide a new view into the biases that make language learning possible in both human and artificial learners. we use transformer language learners and perform causal interventions that alter their inductive learning biases before training them on natural languages. our findings show the relative strengths of different structural inductive biases: simple, regular structure is far outperformed by both contextfree and non-context-free structural relationships, but a recursive structure is not necessarily the best such complex bias. we obtain our results from experimenting on artificial language learners, rather than working with humans or analyzing human language. inductive bias experiments on artificial learners let us easily identify possible hypotheses of structural inductive biases in human cognition. more importantly, since we are working with systems we can influence (rather than analyzing properties and universals of language) we can test hypotheses that are not dependent on linguistic theory-building. we can therefore explore outside the well-studied hypotheses in the linguistics literature. our experiments address questions about the learnability of human languages under different structural inductive biases which have been proposed in the linguistics and cognitive science literature as bases for language. however, experiments such as ours cannot directly prove how the human language system is structurally biased. what such experiments can achieve is to shine light on language as a learnable system as a whole, and produce hypotheses about the relevant structures that could be guiding human language learning. in our case, we find that recursion is indeed a strong inductive bias for language learning. however, our experiments also serve to point out that it's not nesting constituent, structurally-recursive properties that necessarily most help language learning. using the cross language, we show that when we take away recursion from the inductive bias, but keep the overall framework of paired tokens that connect in non-trivial ways, we get a better inductive learning bias. thus, our work serves to showcase possible alternatives to @xcite hypothesis that recursion is a necessary bias: other structurally-complex ways of relating tokens may be just as strong. differences such as those between recursion and non-recursive complex structures are hard to disentangle theoretically. as such, an experimental paradigm such as ours is a helpful step in widely exploring the hypothesis space in questions around human language cognition. artificial models of language can never definitively prove facts about human language processing. the models pretrained and fine-tuned in this paper are products of opaque optimization processes, and the results that we get do not necessarily match human learning. nevertheless, our work serves to identify and examine different hypotheses about human inductive learning biases. we hope that such work can help enrich the hypothesis space over which theoretical argumentation, as well as human subject experiments, are conducted, ultimately leading to a richer understanding of the human language learning process.