How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases (2023)

Abstract: Accurate syntactic representations are essential for robust generalization in natural language. Recent work has found that pre-training can teach language models to rely on hierarchical syntactic features—as opposed to incorrect linear features—when performing tasks after fine-tuning. We test what aspects of pre-training are important for endowing encoder-decoder Transformers with an inductive bias that favors hierarchical syntactic generalizations. We focus on architectural features (depth, width, and number of parameters), as well as the genre and size of the pre-training corpus, diagnosing inductive biases using two syntactic transformation tasks: question formation and passivization, both in English. We find that the number of parameters alone does not explain hierarchical generalization: model depth plays greater role than model width. We also find that pre-training on simpler language, such as child-directed speech, induces a hierarchical bias using an order-of-magnitude less data than pre-training on more typical datasets based on web text or Wikipedia; this suggests that in cognitively plausible language acquisition settings, neural language models may be more data-efficient than previously thought.

## introduction accurate syntactic representations are necessary for robust generalization to new natural language inputs and for the generation of correct outputs. consider the problem of identifying the subject of "said" in the following sentence: can you repeat what the senator next to the cats said? typical language models (lms), which receive linear sequences of words as input, could conceivably rely on a linear or positional feature that usually, but does not always, identifies the correct subject of a verb. an lm could learn, for example, that the first noun in the sentence is always the subject. this heuristic works for many simple sentences, but fails in ex. (1): here, the first noun is "you", and so this heuristic would lead the lm to incorrectly interpret the sentence as meaning "can you repeat what you said?" the lm could also learn that the subject of the verb is the noun closest to the verb in the linear order of the sentence, in which case it would interpret ex. (1) as "can you repeat what the cats said?" by contrast, an lm that represents the sentence as hierarchically structured will correctly identify senator as the subject of the embedded clause that contains the verb said. this example demonstrates that a preference for syntactic features over linear features is required for robust linguistic generalization. the success of large-scale pre-training across fine-tuning tasks suggests that exposure to natural language may teach models to rely on appropriate syntactic features instead of heuristics (even though models still often rely on heuristics; @xcite . this hypothesis is supported by the finding that, given minimal pairs of grammatical and ungrammatical sentences, the probability distribution over sentences defined by lms often favors the grammatical sentence @xcite . a related line of work has shown that, through pre-training, lms can under some circumstances acquire syntactic inductive biases which are then applied to fine-tuning tasks, whereas models which have not been pre-trained do not have such inductive biases @xcite @xcite . when does pre-training endow lms with a syntactic inductive bias? in this study, we address two specific sub-questions: (1) which architectural features make a syntactic inductive bias more likely to emerge in a transformer lm? (2) how is the inductive bias affected by the genre and size of the pre-training corpus? we investigate these questions by evaluating a range of transformer encoder-decoder models based on t5 @xcite . we evaluate both existing publicly available models and models that we pre-train ourselves; we explore a variety of model widths (embedding and hidden dimension, feed-forward layer size) and depths (number of layers), and pre-train on corpora of varying genres and sizes. we then evaluate models' inductive biases by observing their outof-distribution generalization when fine-tuned on syntactic transformations tasks ( §4). we find that depth matters more than width for the acquisition of hierarchical biases ( §5), and that pretraining on simpler language induces hierarchical biases using far less data ( §6 and §7). this last finding suggests that in language acquisition settings in which the training corpus more closely resembles the language that children are exposed to, transformers may be more sample-efficient than previously thought. our code is available on github. foot_0 ## background and motivation every finite training set is consistent with multiple generalizations. we use the term inductive bias to refer to the set of assumptions that a model relies on when generalizing to new data. our usage includes any factor that leads the model to generalize in one way rather than another @xcite ; this can include not only the model's architecture, but also representations learned from prior or concurrent training on tasks that are related to the target task @xcite , and in particular self-supervised pre-training @xcite . we can infer a model's inductive bias by observing how it generalizes out of distribution after training on a dataset that is compatible with multiple generalizations. applying this methodology, @xcite , @xcite , and @xcite find that lstm and transformer encoder-decoder models trained from scratch (without pre-training) on syntactic transformations, such as converting a declarative sentence into a question ( §3), do not generalize in a hierarchical manner. by contrast, @xcite find that certain pre-trained encoder-decoder models-including t5 and bart @xcite )-do generalize hierarchically after fine-tuning. @xcite and @xcite re-port similar results for the pre-trained masked lm roberta @xcite , though in their study a robust syntactic inductive bias only emerged when the training corpus was much larger than a human might be exposed to. previous work on the effect of training corpus size and genre on syntactic generalization includes @xcite , who find that masked lms show stronger syntactic abilities after training on a few million words of child-directed speech than a similar amount of wikipedia or news text; they do not, however, explore whether similar abilities arise from training on a larger amount of wikipedia text. @xcite report experimental results suggesting that scaling the training corpus or model size is unlikely to result in human-like syntactic abilities for lstm lms, but they only vary model width and only train on wikipedia data. we fill the gap between these studies by investigating the influence of multiple component of the transformer architecture and by training on corpora of varying genres and sizes. our work is related more broadly to the syntactic lm evaluation literature. in this style of work, evaluation is typically performed using minimal pairs, where a grammatical and ungrammatical sentence or completion are provided to a model, and the model is expected to assign a higher probability to the grammatical variant. syntactic evaluations have found that lstm- @xcite and transformer-based @xcite lms are sensitive to grammatical number and gender in subject-verb agreement and reflexives @xcite @xcite @xcite . lms are also sensitive to filler-gap dependencies @xcite and, to a lesser, extent, negative polarity items @xcite . this holds across languages @xcite and across grammatical/typological features @xcite . overall, prior work has shown that pre-training can impart hierarchical inductive biases to lms. the goal of this study is to examine which aspects of pre-training-specifically, architecture and training data-contribute to the emergence of this bias. ## syntactic transformations to evaluate the linguistic inductive biases of our models, we employ the poverty of the stimulus design @xcite : we fine-tune a model on ambiguous data and then evaluate it on out-ofdistribution examples where only the desired inductive bias will result in the correct outputs. here, we use the syntactic transformations paradigm @xcite summarized in figure 1 , and observe whether models generalize according to hierarchical linguistic rules or according to surface heuristics based on word position or relative word ordering. we evaluate on english question formation and passivization, using the english datasets of @xcite (themselves based on mccoy et al. 2020). ## overview of experimental paradigm all of our experiments involve fine-tuning variants of t5, a transformer encoder-decoder model pre-trained using a span denoising objective: contiguous token sequences are masked in the input sequence and then reconstructed in the output sequence. we either use the publicly available pretrained "efficient" t5 models released by tay et al. the syntactic transformation datasets we finetune on are the english datasets of @xcite , which consist of 100,000 training examples; 10,000 in-distribution test examples, which test whether the models have learned the task; and 10,000 out-of-distribution generalization examples, which reveal models' inductive biases. we adopt mueller et al.'s hyperparameters (app. a). we fine-tune for 10 epochs (approximately 7500 training steps), and every 500 steps we save a checkpoint and evaluate it. across models, accuracy on the in-distribution test set generally reaches 100% within 500 steps (the first checkpoint) and remains 100% throughout fine-tuning. because in-distribution test set accuracy may not correlate with generalization accuracy, it is unclear which checkpoint would yield the best accuracy on the generalization set; we therefore report the mean generalization accuracy across all checkpoints. ## architectural effects which architectural features contribute to hierarchical generalization? given that language is struc2 the term "efficient" here contrasts the models of @xcite with the original t5 models of @xcite , which are pareto-inefficient with respect to downstream performance and number of parameters. the "efficient" models achieve similar performance across tasks using fewer parameters by using a deeper (more layers) and narrower (smaller hidden size/feed-forward size) architecture. tured hierarchically, we hypothesize that model depth (number of layers) will be the most important component, as deeper structure could more easily allow for representations of deeper hierarchical structures (e.g., more complex syntax trees), with recursive syntactic operations applied successively across layers @xcite . ## corpus genre large lms are typically pre-trained on web text and/or wikipedia data-genres that are distinct from the type of language that humans are exposed to during childhood. could the domain of pretraining corpora explain why lms require much more data than humans to reach similar syntactic abilities @xcite report experiments that support this hypothesis: they find that the roberta masked lm achieves higher accuracies on linguistic acceptability judgment benchmarks when it is pre-trained on child-directed speech as opposed to a similar amount of wikipedia data. in this section, we investigate whether this applies to our paradigm by pretraining encoder-decoder models on child-directed speech and a similar amount of text drawn from the english wikipedia. ## discussion why does depth facilitate the emergence of a syntactic bias? our first set of experiments suggests that depth is the most important architectural factor contributing to hierarchical generalization in transformers. this finding is consistent with the suggestion of @xcite , who advocate for deeper and narrower architectures for the best performance across nlp tasks. why are deeper models better in practice for many tasks and linguistic evaluations, when in theory an arbitrarily wide model can approximate any function with only two layers @xcite ? one natural hypothesis is that transformers generalize hierarchically on the basis of tree-structured representations organized across layers, such that higher layers represent larger constituents, and recursive syntactic operations are applied across successive layers; such a strategy arises more naturally in a deeper model. in recent work, @xcite find evidence that the internal organization of transformer representations across layers becomes more tree-like over the course of training on some tasks, and that this property predicts the model's compositional generalization. while they fail to find a correlation between model depth and the degree to which representations are tree-shaped, this may be because they train relatively small models from scratch on synthetic datasets. in future work, methods such as those of @xcite may be used to measure the tree-likeness of transformers' representations throughout pre-training on natural language, and the degree to which the tree-likeness of the pre-trained model correlates with the its syntactic inductive bias for fine-tuning. why does simpler language teach syntax more effectively? we find that pre-training on simpler language, such as child-directed speech or simple wikipedia, enables hierarchical generalization from far less pre-training data than more complex language. our findings from encoder-decoder models are consistent with previous findings from encoder-only masked lms @xcite , and with work on language understanding from speech @xcite ). the advantage of child-directed speech may be attributable to reduced lexical complexity, reduced syntactic complexity, or both @xcite . lower lexical complexity-in this case, fewer word types-may make it possible to learn the distribution of, say, parts of speech from a smaller corpus, as the same words would recur more often in different contexts. lower syntactic complexity could result in a higher proportion of short sentences with unambiguous syntactic structure, which could help bootstrap syntactic learning. these two features are correlated in natural child-directed speech, but could be disentangled in future work by independently manipulating the lexical and syntactic distributions. simpler language can be leveraged for more efficient pre-training. our experiments show that not all pre-training data is created equal, and motivate further research on data curation for pretraining, and in particular on curriculum learning @xcite . we conjecture that robust syntactic inductive biases will play a role not only in fine-tuning but also in pre-training, making it possible for models to use additional pre-training sentences more efficiently. this motivates a two-phase "starting small" approach @xcite , where the model is first exposed a model to child-directed speech until syntactic inductive biases emerge, and then pre-training on a larger corpus proceeds as usual afterwards. this approach is related to, but distinct from, the single-phase simple-to-complex approach, where a pre-training dataset is sorted from the simplest inputs to the most complex and then presented to a model in order. the singlephase approach has demonstrated mixed results @xcite , but to our knowledge, a syntax-focused two-phase approach has not yet been attempted. transformers may be more data-efficient than previously thought. our findings about the amount of pre-training data required for the acquisition of syntactic biases also have implications for cognitive modeling research. humans learn language from far fewer words than contemporary lms, and at the same time generalize their linguistic knowledge to new settings more robustly; conversely, standard nlp evaluations, which do not take the pre-training corpus into consideration, implicitly reward architectures that learn well from vast amounts of data, raising the concern that those architectures are suboptimal for cognitive modeling (linzen, 2020). our evaluation setup and empirical results go some way towards addressing these concerns: we show that pre-training on a developmentally plausible amount of data can induce human-like inductive biases that improve outof-distribution generalization. this suggests that transformers, when trained in cognitively relevant regimes, may serve as fruitful models of human language acquisition and processing (see also @xcite . ## conclusions we have analyzed the architectural and data features that contribute to the acquisition of syntactic inductive biases during the pre-training of encoderdecoder transformers. we find that model depth matters more for hierarchical generalization than other model components ( §5); that models more quickly learn that language is hierarchical given simpler language ( §6); and that it takes orders-ofmagnitude more data to induce hierarchical inductive biases when pre-training on genres such as wikipedia or web text, compared to simpler data such as child-directed speech ( §7).