Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models (2022)

Abstract: Relations between words are governed by hierarchical structure rather than linear ordering. Sequence-to-sequence (seq2seq) models, despite their success in downstream NLP applications, often fail to generalize in a hierarchy-sensitive manner when performing syntactic transformations—for example, transforming declarative sentences into questions. However, syntactic evaluations of seq2seq models have only observed models that were not pre-trained on natural language data before being trained to perform syntactic transformations, in spite of the fact that pre-training has been found to induce hierarchical linguistic generalizations in language models; in other words, the syntactic capabilities of seq2seq models may have been greatly understated. We address this gap using the pre-trained seq2seq models T5 and BART, as well as their multilingual variants mT5 and mBART. We evaluate whether they generalize hierarchically on two transformations in two languages: question formation and passivization in English and German. We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not. This result presents evidence for the learnability of hierarchical syntactic information from non-annotated natural language text while also demonstrating that seq2seq models are capable of syntactic generalization, though only after exposure to much more language data than human learners receive.

## introduction human language is structured hierarchically. in nlp tasks like natural language inference, syntactic competence is a prerequisite for robust generalization (e.g., @xcite . probing studies have found that masked language models (mlms) contain hierarchical representations @xcite @xcite figure 1 : the poverty of the stimulus experimental design. we fine-tune pre-trained seq2seq models and train small seq2seq models from scratch to perform syntactic transformations. the training set contains ambiguous examples consistent with hierarchical and linear transformation rules. the generalization set contains examples where only the hierarchical rule results in the correct output. pre-trained models generalize using the hierarchical rule, while models trained from scratch generalize using the linear rule. @xcite , while behavioral studies of recurrent neural language models @xcite @xcite and mlms @xcite have found that models are largely able to capture long-range syntactic dependencies that require hierarchical representations of sentences. recent evidence suggests that mlms like bert @xcite and roberta @xcite can learn to make hierarchical linguistic generalizations through exposure to text @xcite , though acquiring many of these linguistic generalizations requires large amounts of data @xcite . however, this evidence comes from binary acceptability judgment tasks, where a classifier head is attached to an mlm and the model is fine-tuned to classify which sentence in a given minimal pair is consistent with a hierarchical linguistic generalization, rather than a positional surface heuristic. consider the following two transformations of example (1): example (1a) correctly forms the question by moving the main auxiliary verb to the front of the sentence, while (1b) relies on the incorrect positional heuristic that the first auxiliary in the declarative sentence should be moved to the front of the sentence. when differentiating grammatical and ungrammatical auxiliary movements, a model could rely on distributional information @xcite ) such as bigram heuristics @xcite to make correct judgments in many cases, so high performance on binary classification tasks may overstate the syntactic competence of a model. by contrast, performing a syntactic transformation-e.g., given a declarative sentence like example (1) as input, transforming it into a polar question like (1a)-is more difficult. it requires multiple complex but systematic operations that rely on hierarchical structure, including movement, number agreement, and-in languages that have grammatical case, such as german-case reinflection. evaluations of syntactic transformational abilities can therefore act as more targeted behavioral indicators of syntactic structural representations in neural models. @xcite evaluate non-pre-trained recurrent sequence-to-sequence (seq2seq) models @xcite on the question formation task, finding that they rely on linear/positional surface heuristics rather than hierarchical structure to perform this syntactic transformation. more recent studies have also exclusively considered recurrent seq2seq models and transformer models @xcite trained from scratch on other transformations like tense reinflection @xcite and passivization @xcite , finding similar results. these studies were designed to understand the inductive biases of various seq2seq architectures, which is why they do not pre-train the models on non-annotated natural language data before training them to perform syntactic transformations. in this study, we create german datasets and modify english datasets for evaluating the induc-tive biases of pre-trained models. we use these datasets to analyze performance in monolingual and zero-shot cross-lingual settings. further, we analyze how pre-trained models perform syntactic transformations. our findings indicate that pretrained models generally perform syntactic transformations in a hierarchy-sensitive manner, while nonpre-trained models (including randomized-weight versions of pre-trained models) rely primarily on linear/positional heuristics to perform the transformations. this finding presents additional evidence to @xcite and @xcite for the learnability of hierarchical syntactic information from natural language text input. our code and data are publicly available. foot_0 2 syntactic transformations ## results all models learn the in-distribution transformations. we first present results on unseen sentences whose structures were seen in training, where both the hierarchical and the linear rules result in correct generalization ( ## transformation strategies our results indicate that pre-trained seq2seq models can consistently perform hierarchy-sensitive transformations. what strategy do they follow to do this? because pre-training corpora include actives, passives, declaratives, and questions, model representations could encode these high-level sentence features. 9 thus, one strategy could be to learn a mapping between abstract representations of different sentence structures (representation strategy). alternatively, models could learn to correctly identify the relevant syntactic units in the input, and then learn a "recipe" of steps leading to the correct transformations (recipe strategy). to distinguish which strategy models use to perform syntactic transformations, we observe crosslingual zero-shot transfer on syntactic transformations. we exploit that english and german use the same operations for question formation, whereas passivization in german involves the additional steps of case reinflection and moving the main verb. if structural representations are shared across english and german, foot_9 we do not expect divergent behaviors for question formation and passivization: if a model employs the representation strategy, then after fine-tuning on only english passivization, it should also correctly perform german passivization, including the additional steps of case reinflection and moving the main verb. conversely, if it employs the recipe strategy, we expect a model trained on english passivization to only perform the steps that are required for english passivization, resulting in incorrect case marking and no main verb movement in german. we first verify that mt5 and mbart are capable of cross-lingual transfer by training a model on the english question formation task and evaluating on german. in early experiments, we noticed the issue of "spontaneous translation" @xcite ; we therefore also include german identity transformations in the training data to train the decoder to also output german sentences. as the top two panels of figure 3 show, mt5 can correctly perform german question formation on in-domain structures (rcs on objects) after being exposed only to english transformations. for out-of-domain structures (rcs on subjects), mt5 almost always moves the main auxiliary but almost never deletes it from its original position, resulting in lower sequence accuracies. apart from this error, the model is capable of cross-lingual transfer on the question formation task. by contrast, mbart achieves poor results on zero-shot german question formation, so we cannot make conclusive arguments using this approach; see appendix d. given that cross-lingual transfer is possible for mt5, how does the model behave in the passivization task, which differs between english and german? we fine-tune mt5 on english passivization (as well as german identity transformations on active sentences). the results of this experiment (the lower two panels in figure 3 ) show that the model is still able to move the main object to the subject position, but also that it never correctly performs german passivization in its entirety. this is because the model performs exactly the same steps for german sentences as for english sentences, resulting in outputs with english syntax: these behavioral patterns suggest that mt5 employs the recipe strategy: it succeeds if a transformation's required operations are the same across languages (as for question formation) but fails if the steps differ (as for passivization). even in passivization, however, the model still learns to move the correct nps, which provides additional evidence that mt5 makes use of structural features when performing transformations. given the similarities between mt5's and mbart's architectures and training setups, one could reasonably presume that mbart may follow a similar strategy to perform syntactic transformations; nonetheless, mbart is less consistent in performing syntactic transformations, so this method cannot present strong evidence for use of the recipe strategy for that model. ## corpus analysis pre-trained models learn to use hierarchical features for performing syntactic transformations. is this because there is explicit supervision for the hierarchical rules in the pre-training corpora? in other words, are there disambiguating examples in these models' training corpora that helps them memorize hierarchical transformation patterns? here, we focus on english question formation examples in mt5's training corpus. 11 disambiguating examples would be rare, as a single pre-training context window must contain a declarative sentence as well as the same sentence transformed into a question; humans would tend to replace at least some of the constituents with pronouns or delete them (e.g., @xcite . it would also require the move-first rule to not correctly transform the sentence-and for at least one of the auxiliaries to be noised in one sentence but not the other, such that the auxiliary has to be recovered from the other sentence. for example: we search for english disambiguating question formation examples. to this end, we sample 5m english documents from mt5's training corpus mc4, segmenting each document into sentences using spacy. foot_10 this yields 118.3m sentences. we examine each pair of adjacent sentences in each document, manually inspecting any sentence pair meeting the following criteria: (1) the token jaccard similarity of the sentences @xmath0.7; (2) one sentence begins with an auxiliary verb and the other does not; (3) there are at least two distinct auxiliaries in both sentences. there are 277 sentence pairs in our sample that met all criteria, of which 13 are adjacent declarative/question pairs that are equivalent except for the fronted auxiliary. thus, the probability of an equivalent declarative/question pair with two auxiliaries in mc4 is ≈ 1.1 × 10 -7 . as t5's and mbart's training corpora consist of data from similar webtext distributions, it is likely that these structures exist in those corpora as well. crucially, however, none of the declarative/question pairs were disambiguating examples: each pair was consistent with the linear move-first rule. what is the probability of a disambiguating example, then? if we assume that the probability of a sentence containing an rc on the subject is independent from the probability of a declarative/question sentence pair, we can take the product of both probabilities to obtain an estimate. from the same sample of 118.3m sentences, we use spacy's dependency parser to extract sentences containing an rc on the subject and where at least one auxiliary verb appears in the sentence. we obtain 526, 944 such sentences, meaning that the probability of an rc on a subject in an auxiliary-containing sentence in mc4's english corpus is ≈ 4.5 × 10 -3 . thus, the probability of declarative/question pair with an rc on the subject and auxiliary in the rc is ≈ (4.5×10 -3 )•(1.1×10 -7 ) = 4.95×10 -10 . mt5 is trained on up to 1t tokens of data, and 5.67% of its documents are english; it therefore observes ≈ 56.7b english tokens. if we optimistically assume that english sentences contain an average of 15 tokens, it observes 3.78b english sentences. then we would expect 3.78b×(4.95 × 10 -10 ) ≈ 2 disambiguating examples. this is not including the auxiliary masking criterion, which would make such examples even less likely. thus, while we cannot definitively rule out the possibility of disambiguating examples in mc4, they are rare if they exist in the corpus at all. nonetheless, we have found evidence for supervision on question formation in the form of adjacent declarative/question sentence pairs, even if they do not explicitly support the hierarchical rule. ## discussion our experiments provide evidence that pre-trained seq2seq models acquire a hierarchical inductive bias through exposure to non-annotated natural language text. this extends the findings of warstadt and bowman (2020) and @xcite to a more challenging generative task, where models cannot rely on n-gram distributional heuristics @xcite . this also provides additional evidence that masking and reconstructing subsets of input sequences is a powerful training objective for inducing linguistic generalizations, whether in masked language models like roberta @xcite or sequence-tosequence models. span denoising ((m)t5's objective) appears more effective for learning syntactic transformations than full sequence reconstruction ((m)bart's objective) given that (m)t5 is more consistently able to perform transformations, though there are too many other differences in training data and hyperparameters between (m)t5 and (m)bart for us to be able to directly implicate the training objective. this hypothesis can be tested explicitly in future work by training identical models that differ only in their pre-training objective. counter to @xcite , our findings suggest that hierarchical architectural constraints (e.g., tree-structured networks) are not necessary for robust hierarchical generalization as long as the model has been exposed to large amounts of natural language text-possibly far more language than humans would be exposed to. however, one difference between the randomly initialized models employed by @xcite and pre-trained models is that pre-trained models have likely seen the structures (but not sentences) present in the generalization set; thus, rather than relying on syntactic features, the model could choose the correct transformation because it is more similar to the grammatical examples it has already seen. we found declarative/question pairs in mt5's training corpus, but we did not find any examples that explicitly demonstrated the hierarchical rule for question formation. while we cannot fully rule out the possibil-ity of disambiguating examples, this strategy is still unlikely given that pre-trained models produce ungrammatical transformations, both in monolingual transformations (e.g., not deleting the main auxiliary after copying it to the start of the sentence) and in cross-lingual german passivization. additionally, because we use greedy decoding, models are not able to take future words into account when predicting the fronted auxiliary: they must select the appropriate auxiliary to move solely based on the encoder's representations. more broadly, our findings counter the assumption that a hierarchical constraint is necessary in language learners to acquire hierarchical generalization @xcite . while the pre-trained models that we considered observe far more input than a child would receive @xcite , @xcite recently demonstrated high performance on grammaticality judgments for models trained on much smaller child-directed speech corpora, suggesting that our findings may also hold when training models on more human-like input. ## conclusions we have performed an analysis of the syntactic transformational ability of large pre-trained sequence-to-sequence models. we find that pretrained models acquire a hierarchical inductive bias during pre-training, and that the architecture does not yield this hierarchical bias by itself. it remains an open question whether such deep and highly parameterized models or such large pretraining datasets are necessary for hierarchical generalization. future work could ablate over model depth and pre-training corpus size to observe the relative contribution of architecture and the training set to inducing hierarchical inductive biases in seq2seq models.