Recurrent babbling: evaluating the acquisition of grammar from limited input data (2020)

Abstract: Recurrent Neural Networks (RNNs) have been shown to capture various aspects of syntax from raw linguistic input. In most previous experiments, however, learning happens over unrealistic corpora, which do not reflect the type and amount of data a child would be exposed to. This paper remedies this state of affairs by training an LSTM over a realistically sized subset of child-directed input. The behaviour of the network is analysed over time using a novel methodology which consists in quantifying the level of grammatical abstraction in the model’s generated output (its ‘babbling’), compared to the language it has been exposed to. We show that the LSTM indeed abstracts new structures as learning proceeds.

## related work a considerable amount of literature has investigated the ability of anns to acquire grammar, and the list we present here is by no means exhaustive. the analysis of the syntactic abilities of lstms @xcite and ann-based language models dates back quite a few years @xcite ). recent contributions have followed a general tendency to analyze the inner-workings of networks, and the specific type of knowledge they acquire @xcite . for instance, @xcite show how a network acquires abstract information about number agreement, albeit in a supervised setting. the same study is expanded in @xcite , which shows how a language modeling task is enough for a network to predict long-distance number agreement, both on semantically sound and nonsensical sentences. the authors conclude that "lm-trained rnns can construct abstract grammatical representations", but their model is trained on a rather consequent amount of data (90m tokens) from a rather peculiar distribution (a wikipedia snapshot). similarly, it has been shown that lstms @xcite can learn tricky syntactic rules like the english auxiliary inversion and filler-gap dependencies, although, in later work, @xcite find that only models with an explicit inductive bias @xcite learn to generalize the move-main rule with respect to auxiliary inversion. @xcite show instead poor performance of rnns in grammaticality evaluation, due to their sensitivity to the specific lexical items encountered during training, a limitation that, they say, "would not be expected if its syntactic representations were fully abstract". similarly chowdhury and zamparelli (2018) state that their model "is sensitive to linguistic processing factors and probably ultimately unable to induce a more abstract notion of grammaticality". moreover, despite the fact that the model of @xcite is tested on four languages, the most promising results may not be generalizable to languages showing different surface patterns from english. @xcite fail to replicate @xcite 's results on basque, and @xcite , after testing the network on relative clause attachment cases in english and spanish, conjecture that the associative (non-linguistic) bias of rnns overlaps with english syntactic structure but represents an obstacle to learn attachment rules for spanish. other puzzling results concern the relation of perplexity to syntactic performance @xcite : having evaluated their models on 34 benchmarks, @xcite conclude with a call for a wider variety of syntactic phenomena to test on. further studies have shown that networks carrying explicit inductive bias perform better than vanilla lstms. in a recent paper, @xcite show that a constituency-based network generalizes more robustly than a dependencybased one, and that both outperform a more basic bilstm. lastly, we mention the study carried out by @xcite who perform their study using a character-based lstm -a choice we will follow in this work. a very similar scientific discussion, which we won't report in depth here, is blooming around transformer-based language models @xcite @xcite @xcite , leading to similar contrasting results. finally, a separate line of work focuses on a more indirect test of the information encoded in the internal representation, assessing which aspects of the original syntactic structure can be reconstructed through diagnostic classifiers @xcite @xcite . in summary, a clear trend has not yet emerged @xcite . all the models we cited, however, seem to idealize syntactic structure as a separate and more abstract ability from the knowledge of statistical regularities or lexical co-occurrences. this perspective may reflect a belief in a sharp distinction between the lexicon and compositional rules. that is, anns are expected to gain abstract grammatical abilities through compositional generalization, where compositionality is understood as the ability to produce an unbounded number of sentences by means of a set of algebraic rules @xcite . in contrast with this approach, usage-based models encourage us to adopt a different perspective, and to analyze lstms' grammatical abilities with respect to the kind of representations (more in §3.3) posited by theories such as construction grammar @xcite @xcite . ## framework in essence, the question of language acquisition asks how much language (λ) can be learned with a certain level of computational complexity (c) by being exposed to a certain type of data (i). the corresponding formalization, a : c × @xmath0, describes both human and artificial acquisition processes, and its components have been central in the linguistic debate. below, we will discuss each term (c, i and λ) in further detail. ## meaning and abstraction our second analysis relies on the idea that we can state that the network has learned some grammar once it is able to use an acquired pattern in a productive and creative way. following the basic hypothesis of cxg, stated in § 3.3, we expect this generalization ability to evolve during training and the distributional properties of patterns to be in relation with the grammatical abilities of the network at various stages of learning. let's consider the structures cat 1 : the dog and cat 2 : det noun. for the purpose of our analysis, we will consider (cat 1 , cat 2 ) to be a minimal pair, as the dog can be considered a lexicalized instance of the more abstract construction det noun. using a distributional analysis, we can capture how the contexts of cat 1 and cat 2 vary, and how this variation is associated with generalization. if their cosine similarity decreases during training, it means that their contexts become more and more dissimilar: the network produces det noun in new contexts which do not perfectly overlap with those of the dog, indicating that the network's babbling is becoming more productive (a graphical representation is given in figure 5 ). in this case, we theorize that cat 2 has been recognised as a partially independent pattern from cat 1 . if, on the contrary, their cosine similarity increases, we might deduce that the network has recognized cat 2 as partly unnecessary: it is correcting an overgeneralization. we restrict this analysis to the childes corpus. we build distributional vector spaces for the input and each stage of babbling using the dissect toolkit @xcite . we consider catenae figure 5 : let us assume that the input presents various lexicalized instances of the pattern det noun (e.g. the dog, the cat, a giraffe). our hypothesis is that the network will only be able to capture its more stereotypical instances (i.e., the dog), and the distributions of the dog and det noun will thus almost perfectly overlap in the first stages of babbling (the length of vectors in the figure is just for exemplification). at later stages, the language produced by the network will show greater traits of productivity: the distribution of det noun might show that its cosine distance to the dog has increased as it is now instantiated by two different lexicalized patterns (the dog and the cat) that are produced in dissimilar contexts. we then compute average distributional shifts and cosine similarities, grouping all pairs by cat 1 and cat 2 values (for instance, we average all pairs that show abstractions of cat 1 : a minute, as well as pairs that show instantiations of cat 2 : det noun). some averages are shown in we finally split catenae in three bins based on average distributional shift and investigate the influence of input similarity over the abstraction behaviour of a construction. our hypothesis is that catenae that underwent the highest shifts during training were those showing intermediate levels of similarities in the input distributional space. indeed, pairs with very high input similarities are unlikely to exhibit abstraction: according to constructionist intuition, their distributional similarity means that the catena that is part of the constructicon is the least abstract one, and there is no need for the more abstract category. low similarity pairs, on the other hand, may simply contain unrelated catenae. to test our hypothesis, we perform a kruskallwallis one-way analysis of variance test, that turn out to be significant for groupings made on both cat 1 and cat 2 lists. 10 the result is confirmed by dunn's posthoc test. we show results for the test performed on the cat 2 list in usage-based computational accounts have already shown to be able to explain puzzling phenomena in acquisition @xcite or to induce syntactic rules in an unsupervised manner @xcite , making use of surface properties of the language signal like transitional probabilities or basic distributional analysis. however, despite being rooted in the psychological literature and yielding fundamental psycholinguistic results, the models presented in such investigations are often not comparable to studies involving neural language models, as the former are usually less flexible and less scalable to large amounts of data than the latter. in this paper, we have reviewed relevant work concerning the assessment of grammatical abilities in neural language models and noted the lack of variety in both the input data fed to anns (i) and the theoretical framework used in analysing the output language (λ). in line with the existing usagebased computational accounts, we have introduced a methodology to evaluate the level of productivity of an lstm trained on limited, child-directed data, using inspirations from constructionist approaches. we have been able to show that neural networks approximate the distribution of constructions at a quite refined level when trained over a bare 3m most importantly, the introduced methodology, despite being preliminary, presents a number of features that make our study fit in the usage-based theoretical framework while also using neural networks as language modeling tools, more specifically: (i) it posits no sharp distinction between lexicon and grammar: fully lexicalized, partially filled and purely syntactic patterns are all part of our constructicon and can play a similar role in production. different items can therefore be represented compared, irrespective of their lexical nature; (ii) it makes no assumption about the stability of the constructicon: what is relevant for productivity at the earliest stages of learning might become superfluous later on; (iii) all items are seen as formmeaning pairs (i.e., constructions by definition, as in @xcite : a novel way of modeling constructional meaning is therefore introduced and represents a promising path for future studies; (iv) distributional semantics is used both as a powerful quantitative tool and as a usage-based cognitive hypothesis, which leads us to specific assumptions about the cognitive format and origin of semantic representations @xcite , and seems in line with the view of constructions as "invitations to form categories" @xcite . finally, we must account for potential biases introduced by applying dependency parsing to both input data and neural babbling: while this step is necessary to extract catenae, it introduces a non-negligible amount of noise, as the available pipelines are typically trained on different varieties than the ones considered in this study. in particular, the parser is somehow projecting its own categories, which have been acquired in a different setting and probably on a different variety, on our data. this currently limits the transferability of our results. besides looking for ways to circumvent this issue, further work includes a comparison of our results with a wider choice of models.