Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks (2020)

Abstract: Learners that are exposed to the same training data might generalize differently due to differing inductive biases. In neural network models, inductive biases could in theory arise from any aspect of the model architecture. We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection. For both tasks, the training set is consistent with a generalization based on hierarchical structure and a generalization based on linear order. All architectural factors that we investigated qualitatively affected how models generalized, including factors with no clear connection to hierarchical structure. For example, LSTMs and GRUs displayed qualitatively different inductive biases. However, the only factor that consistently contributed a hierarchical bias across tasks was the use of a tree-structured model rather than a model with sequential recurrence, suggesting that human-like syntactic generalization requires architectural syntactic structure.

## introduction any finite training set is consistent with multiple generalizations. therefore, the way that a learner generalizes to unseen examples depends not only on the training data but also on properties of the learner. suppose a learner is told that a blue triangle is an example of a blick. a learner preferring shape-based generalizations would conclude that blick means ''triangle,'' while a learner preferring color-based generalizations would conclude that blick means ''blue object'' @xcite . factors that guide a learner to choose one generalization over another are called inductive biases. what properties of a learner cause it to have a particular inductive bias? we investigate this question with respect to sequence-to-sequence neural networks @xcite . as a test case for studying differences in how models generalize, we use the syntactic task of english question formation, such as transforming (1a) into (1b): following @xcite empirical claims about children's linguistic input, we constrain our training set to be consistent with two possible rules illustrated in figure 1 : move-main (a rule based on hierarchical syntactic structure) and move-first (a rule based on linear order). we then evaluate each trained model on examples where the rules make different predictions, such as (2): given (2a), move-main would generate (2b) while move-first would generate (2c): since no such examples appear in the training set, a model's behavior on them reveals which rule the model is biased toward. this task allows us to study a particular bias, namely, a bias for hierarchical generalization, which is important for models of to test which models have a hierarchical bias, we use the question formation task and a second task: tense reinflection. for both tasks, our training set is ambiguous between a hierarchical generalization and a linear generalization. if a model chooses the hierarchical generalization for only one task, this preference is likely due to taskspecific factors rather than a general hierarchical bias. on the other hand, a consistent preference for hierarchical generalizations across tasks would provide converging evidence that a model has a hierarchical bias. we find that all the factors we tested can qualitatively affect how a model generalizes on the question formation task. these factors are the type of recurrent unit, the type of attention, and the choice of sequential vs. treebased model structure. even though all these factors affected the model's decision between move-main and move-first, only the use of a treebased model can be said to impart a hierarchical bias, since this was the only model type that chose a hierarchical generalization across both of our tasks. specific findings that support these general conclusions include: overall, we conclude that many factors can qualitatively affect a model's inductive biases, but human-like syntactic generalization may require specific types of high-level structure, at least when learning from text alone. ## tree models so far we have tested whether properties that are not interpretably related to hierarchical structure nevertheless affect how a model generalizes on a syntactic task. we now turn to a related but opposite question: when a model's design is meant to give it a hierarchical inductive bias, does this design succeed at giving the model this bias? ## tense reinflection we have shown that several models reliably preferred move-main over move-first. however, this behavior alone does not necessarily mean that these models have a hierarchical bias, because a preference for move-main might arise not from a hierarchical bias but rather from some taskspecific factors such as the prevalence of certain n-grams @xcite . a true hierarchical bias would lead a model to adopt hierarchical generalizations across training tasks; by contrast, we hypothesize that other factors (such as a bias for focusing on n-gram statistics) will be more sensitive to details of the task and will thus be unlikely to consistently produce hierarchical preferences. to test the robustness of the hierarchical preferences of our models, then, we introduce a second task, tense reinflection. ## will models generalize across transformations? each experiment discussed so far involved a single linguistic transformation. by contrast, humans acquiring language are not exposed to phenomena in isolation but rather to a complete language encompassing many phenomena. this fact has been pointed to as a possible way to explain hierarchical generalization in humans without needing to postulate any innate preference for hierarchical structure. while one phenomenon, such as question formation, might be ambiguous in the input, there might be enough direct evidence among other phenomena to conclude that the language as a whole is hierarchical, a fact which learners can then extend to the ambiguous phenomenon @xcite , under the nontrivial assumption that the learner will choose to treat the disparate phenomena in a unified fashion. while our training sets are ambiguous with respect to whether the phenomenon underlying the mapping is structurally driven, they do contain other cues that the language is more generally governed by hierarchical regularities. first, certain structural units are reused across positions in a sentence; for example, prepositional phrases can appear next to subjects or objects. such reuse of structure can be represented more efficiently with a hierarchical grammar than a linear one. second, in the question formation task, subject-verb agreement can also act as a cue to hierarchical structure: for example, in the sentence my walrus by the yaks does read, the inflection of does depends on the verb's hierarchically determined subject (walrus) rather than the linearly closest noun (yaks). 9 for the sequential rnns we have investigated, it appears that these indirect cues to hierarchical structure were not sufficient to guide the models towards hierarchical generalizations. however, perhaps the inclusion of some more direct evidence for hierarchy would be more successful. figure 10 : multi-task learning results for a gru without attention. single-task reports baselines from training on a single ambiguous task. multi-task reports results from adding an unambiguous second task. multi-task + auxiliaries reports results from adding an unambiguous second task and also adding overt auxiliaries to the tense reinflection sentences. the numbers give the generalization set performance on the ambiguous task. to take a first step toward investigating this possibility, we use a multi-task learning setup, where we train a single model to perform both question formation and tense reinflection. we set up the training set such that one task was unambiguously hierarchical while the other was ambiguous between the hierarchical generalization and the linear generalization. this gave two settings: one where question formation was ambiguous, and one where tense reinflection was ambiguous. we trained 100 instances of a gru without attention on each setting and assessed how each model generalized for the task that was ambiguous. for both cases, generalization behavior in the multi-task setting differed only minimally from the single-task setting (figure 10 ). one potential explanation for the lack of transfer across tasks is that the two tasks operated over different sentence structures: the question formation sentences always contained overt auxiliaries on their verbs (e.g., my walrus does giggle), while the tense reinflection sentences did not (e.g., my walrus giggles). to test this possibility, we reran the multi-task experiments but with overt auxiliaries added to the tense reinflection sentences (figure 10 , ''multitask + auxiliaries'' row). in this setting, the model still generalized linearly when it was question formation that was ambiguous. however, when it was tense reinflection that was ambiguous, the model generalized hierarchically. we hypothesize that the directionality of this transfer is due to the fact that the question formation training set includes unambiguous longdistance subject-verb agreement as in (13), which might help the model on generalization-set examples for tense reinflection such as example ( 14 by contrast, the tense reinflection training set does not contain any outputs of the type withheld from the question formation training set. if this explanation is correct, it would mean that the improvement on the tense reinflection task derived not from the question formation transformation but rather from the subject-verb agreement incidentally present in the question formation dataset. therefore, even the single potential case of generalization across transformations is likely spurious. recent nlp work has also found that neural networks do not readily transfer knowledge across tasks; e.g., pretrained models often perform worse than non-pretrained models @xcite . this lack of generalization across tasks might be due to the tendency of multi-task neural networks to create largely independent representations for different tasks even when a shared representation could be used @xcite . therefore, to make cross-phenomenon generalizations, neural networks may need to be given an explicit bias for sharing processing across phenomena. ## discussion we have found that all factors we tested can qualitatively affect a model's inductive biases but that a hierarchical bias-which has been argued to underlie children's acquisition of syntax-only arose in a model whose inputs and computations were governed by syntactic structure.