How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech (2023)

Abstract: When acquiring syntax, children consistently choose hierarchical rules over competing non-hierarchical possibilities. Is this preference due to a learning bias for hierarchical structure, or due to more general biases that interact with hierarchical cues in children’s linguistic input? We explore these possibilities by training LSTMs and Transformers - two types of neural networks without a hierarchical bias - on data similar in quantity and content to children’s linguistic input: text from the CHILDES corpus. We then evaluate what these models have learned about English yes/no questions, a phenomenon for which hierarchical structure is crucial. We find that, though they perform well at capturing the surface statistics of child-directed speech (as measured by perplexity), both model types generalize in a way more consistent with an incorrect linear rule than the correct hierarchical rule. These results suggest that human-like generalization from text alone requires stronger biases than the general sequence-processing biases of standard neural network architectures.

## introduction syntax is driven by hierarchical structure, yet we typically encounter sentences as linear sequences of words. how do children come to recognize the hierarchical nature of the languages they acquire? some argue that humans must have a hierarchical inductive bias-an innate predisposition for hierarchical structure @xcite ). an alternative view (e.g., @xcite is that no such bias is necessary: there may be clear evidence for hierarchical structure in children's input, so that children would choose hierarchical rules even without a hierarchical bias. at first blush, recent work in natural language processing (nlp) may seem to indicate that no hierarchical bias is necessary. neural networks trained on naturally-occurring text perform impressively on syntactic evaluations even though they have no explicit syntactic structure built into them (e.g., @xcite @xcite . however, these results do not provide strong evidence about the learning biases required to learn language from the data available to humans because these models receive very different training data than humans do @xcite . first, nlp models are typically trained on far more data than children receive, so models have more opportunities to encounter rare syntactic structures @xcite . second, most training sets in nlp are built from internet text (e.g., wikipedia), which differs qualitatively from the utterances that children typically hear; e.g., sentences in wikipedia are on average 25 words long @xcite , compared to 5 words for sentences in the north american english subset of the childes corpus of child-directed speech @xcite . in this work, to evaluate if neural networks without a hierarchical bias generalize like children do, we train models on text foot_0 comparable to the sentences in children's linguistic input: english data from childes. we then analyze what they have learned about the relationship between declarative sentences, such as (1a), and their corresponding yes/no questions, such as (1b): crucially, nearly all naturally-occurring yes/no questions are consistent with two rules: one based on hierarchical structure (2), and one based on linear order (3): 2,3 (2) hierarchicalq: the auxiliary at the start of a yes/no question corresponds to the main auxiliary of the corresponding declarative. despite the scarcity of evidence disambiguating these rules, children reliably favor hierarchi-calq @xcite , albeit with occasional errors consistent with linearq @xcite . yes/no questions thus are a prime candidate for an aspect of english syntax for which human-like generalization requires a hierarchical bias. we evaluate yes/no question performance in lstms and transformers, two neuralnetwork architectures that have no inherent hierarchical inductive bias @xcite . these architectures employ different computational mechanisms, so consistent results across both would indicate that our results are not due to idiosyncrasies of one particular architecture. to investigate if models generalize more consistently with the hierarchical or linear rule, we evaluate them on cases where the rules make different predictions, such as (4): under hierarchi-calq, the question that corresponds to ( 4a ) is (4b), whereas under linearq it is (4c). we find that across several ways of framing the learning task, models fail to learn hierarchi-calq. instead, they generalize in ways that depend on linear order and on the identities of specific words. these results suggest that children's training data, if taken to be words alone, may not contain enough hierarchical cues to encourage hierarchical generalization in a learner without a hierarchical bias. thus, explaining human acquisition of syntax may require postulating that humans have stronger inductive biases than those of lstms and 2 background though hierarchicalq and linearq often make the same predictions, the evidence in children's input may still favor hierarchicalq. the most straightforward evidence would be utterances that directly disambiguate the rules, such as (4b). @xcite show that disambiguating examples appear in the wall street journal, in literature, and arguably in child-directed speech, but direct evidence may still be too rare to robustly support hierarchicalq @xcite . nonetheless, children might conclude that yes/no questions obey hierarchi-calq rather than linearq based on indirect evidence-evidence that other syntactic phenomena are hierarchical @xcite . to test if the cues favoring hierarchicalq render a hierarchical bias unnecessary, we study how well non-hierarchically-biased models acquire english yes/no questions. several prior papers have used this approach, but their training data differed from children's input in important ways: some used synthetic datasets @xcite @xcite , others used massive internet corpora @xcite , and those that used child-directed speech simplified the data by replacing each word with its part of speech @xcite . we used training data closer to children's input, namely sentences from childes with word identities preserved, rather than being converted to parts of speech. two other recent works have also trained neural networks on childes data @xcite , but neither investigated yes/no questions. one particularly important reason for training models on childes is that, in prior work, different types of training data have yielded diverging results: recent models trained on synthetic data failed to properly acquire yes/no questions @xcite , whereas ones trained on large internet corpora scored well on evaluations of yes/no questions @xcite . given these differing results, it is not clear from past work how these models would generalize when faced with the type of data that children receive. ## overview of experimental setup we evaluated models on yes/no questions in two ways. first, we used relative acceptability judgments (experiment 1): we trained neural networks on the task of language modeling (predicting the next word at every point in the sentence) and evaluated whether they assigned a higher probability to sentences consistent with linearq or hierar-chicalq. our second approach was based on text generation (experiment 2): we trained networks to take in a declarative sentence and output the corresponding question, and tested whether they generalized in a way more consistent with lin-earq or hierarchicalq. under both framings, we trained models on data from childes and evaluated them on targeted datasets constructed to differentiate linearq and hierarchicalq. ## experiment 2: question formation the previous experiment was designed to operate entirely in the next-word-prediction paradigm, motivated by arguments from past literature about the strength and relative ecological validity of next-word-prediction as a training objective (see section 4.2). however, one of this setup's shortcomings is that hierarchicalq describes correspondences between questions and declaratives, but experiment 1 focused on questions alone, with no consideration of declaratives. in this second experiment, to better capture that hierarchicalq is defined over sentence pairs, we trained models on a sentence-pair task: transforming a declarative into a question @xcite . for instance, given the child did learn the model must produce did the child learn ? we evaluated models in two ways. first, we checked if the models' predictions fully matched the correct questions. this full-sentence evaluation is demanding, and models might fail this evaluation for reasons unrelated to our core hypotheses. for instance, given the child did learn the model might produce did the baby learn, which would be marked as incorrect, even though this lexical error is not relevant to hierarchicalq. as a metric that is less demanding and that also more directly targets hierarchicalq, we measured if the first word of the output question corresponded to the first or main auxiliary of the input. critically, linearq and hierarchicalq make different predictions for the first word of a question so long as the two auxiliaries are distinct: see (4). because this framing lets the model freely generate its output (instead of choosing one option from a pre-specified set), we allow for the possibility that the rule learned by models may not be identical to any of our manually-generated hypotheses. solely training models to perform this transformation involves the implicit assumption that, when children acquire english yes/no questions, the only evidence they leverage is english yes/no questions. however, other types of sentences may also provide useful evidence @xcite : e.g., whquestions also illustrate subject-auxiliary inversion @xcite , while, more generally, many types of sentences could provide evidence that syntax as a whole is hierarchical @xcite . to explore this possibility, we compared a condition in which models were only trained to perform question formation (the question for-mation condition) to another in which models were first pre-trained on next-word prediction with the exact same setup as in experiment 1 before being further trained to perform question formation (the next-word prediction + question formation condition). ## discussion we have found that, when trained on child-directed speech, two types of standard neural networks performed reasonably well at capturing the statistical properties of the dataset, yet their handling of english yes/no questions was more consistent with a linear rule linearq than the correct hierarchical rule hierarchicalq. these results support the hypothesis that a learner requires a hierarchical bias to consistently learn hierarchical rules when learning from the linguistic data children receive. ## conclusion in this work, we trained two types of neural networks (lstms and transformers) on sentences of the types available to children and then analyzed what they had learned about english yes/no questions. across several evaluation paradigms, these models failed to generalize in human-like ways: humans display hierarchical generalization, while the models' generalization was instead based on linear order and individual words' identities. our results support the hypothesis that human-like linguistic generalization requires biases stronger than those of lstms and transformers. future work should investigate what inductive biases enable successful generalization. one approach would be to test architectures with built-in hierarchical structure; past work has shown that such architectures have a hierarchical bias @xcite and generalize better on the hierarchical phenomenon of subject-verb agreement @xcite , so they may also generalize better on english yes/no questions. a final direction would be to expand the input beyond words alone so that learners can leverage hierarchical structure that is present in other modalities, such as hierarchical structure in visual scenes.