Modelling Language Acquisition through Syntactico-Semantic Pattern Finding (2023)

Abstract: Usage-based theories of language acquisition have extensively documented the processes by which children acquire language through communicative interaction. Notably, Tomasello (2003) distinguishes two main cognitive capacities that underlie human language acquisition: intention reading and pattern finding. Intention reading is the process by which children try to continuously reconstruct the intended meaning of their interlocutors. Pattern finding refers to the process that allows them to distil linguistic schemata from multiple communicative interactions. Even though the fields of cognitive science and psycholinguistics have studied these processes in depth, no faithful computational operationalisations of these mechanisms through which children learn language exist to date. The research on which we report in this paper aims to fill part of this void by introducing a computational operationalisation of syntactico-semantic pattern finding. Concretely, we present a methodology for learning grammars based on similarities and differences in the form and meaning of linguistic observations alone. Our methodology is able to learn compositional lexical and item-based constructions of variable extent and degree of abstraction, along with a network of emergent syntactic categories. We evaluate our methodology on the CLEVR benchmark dataset and show that the methodology allows for fast, incremental and effective learning. The constructions and categorial network that result from the learning process are fully transparent and bidirectional, facilitating both language comprehension and production. Theoretically, our model provides computational evidence for the learnability of usage-based constructionist theories of language acquisition. Practically, the techniques that we present facilitate the learning of computationally tractable, usage-based construction grammars, which are applicable for natural language understanding and production tasks.

## introduction usage-based theories of language acquisition argue that the ability of children to learn language is based on two general cognitive capacities: intention reading and pattern finding @xcite . intention reading refers to the capacity of children to understand the communicative intentions of their interlocutors. pattern finding refers to the ability to recognise similarities and differences in sensory-motor experiences, and to use this ability for categorisation and schema formation @xcite . pattern finding thus provides mechanisms for generalising across different communicative interactions, thereby constructing abstract schemata that represent the linguistic knowledge of a language user. in the context of language acquisition, intention reading and pattern finding are two key cognitive capacities that are highly complementary. intention reading allows a language learner to reconstruct the meaning of an utterance that they observe during a communicative interaction. pattern finding then provides the mechanisms to learn a grammar based on the combination of these observed utterances and their reconstructed meanings. there exists an impressive body of theoretical and empirical evidence for both intention reading @xcite @xcite and pattern finding @xcite @xcite . however, no comprehensive mechanistic models that provide a faithful operationalisation of either of these cognitive processes exist to date. in this paper, we aim to fill part of this void by presenting a computational operationalisation of pattern finding mechanisms that can bootstrap a grammar based on a set of semantically annotated utterances alone. as such, we assume that the outcome of the intention reading process is given, hence the availability of the utterances' semantic representations, but that neither a segmentation of the utterances nor any pre-existing morpho-syntactic or other grammatical information can be used. for a computational model that operationalises the intention reading process, and that integrates it with the pattern finding mechanisms introduced in this paper, we refer the interested reader to @xcite . a validation of our methodology on the clevr benchmark dataset for visual question answering @xcite shows that it allows for fast, incremental and effective grammar learning. the result of this learning process is a fully-operational, productive construction grammar that can be used for both language comprehension, i.e. mapping from utterances to their meaning representation, and language production, i.e. mapping from a meaning representation to an utterance. the scientific contribution of this paper is twofold. on the one hand, it provides computational evidence for the cognitive plausibility of usage-based theories of language acquisition by introducing a mechanistic model of the acquisition of construction grammars from scratch. on the other hand, the techniques that we present pave the way for learning computationally tractable, large-scale, usage-based construction grammars that facilitate both language comprehension and production. apart from their theoretical importance, such grammars are also highly valuable for a large range of application domains, including intelligent conversational agents @xcite and the semantic analysis of discourse @xcite . the remainder of this paper is structured as follows. section 2 presents the dataset, task and learning problem that we address. section 3 introduces our novel methodology for learning construction grammars. section 4 presents the evaluation results. related work is discussed in section 5. a concluding discussion is provided in section 6. ## data there are two main requirements for datasets to be compatible with the methodology that we present in this paper. first of all, they need to consist of utterances that are annotated with a representation of their meaning. second, they need to be large enough so that they contain enough utterances that are similar to each other, but not equal, in terms of either form or meaning. the availability of ex-emplars that are sufficiently close to each other is a necessary precondition for any generalisation process and is fully consistent with the prevailing hypotheses of how children acquire language @xcite . the exact required size of a dataset is as a consequence directly related to the variety and the degree of complexity of the utterances and meaning representations that it contains. in this paper, we present and validate our methodology using the clevr dataset for visual question answering @xcite . the utterances in the dataset are semantically annotated and the dataset contains ample examples of utterance-meaning pairs that are similar but not equal to each other. the utterances are english questions about images of scenes depicting different configurations of geometrical figures. each question is annotated with a semantic representation that captures the logical meaning that underlies it. an example of such a scene, a question and its semantic representation is shown in figure 1 . the semantic representation in figure 1 takes the form of a set of predicates that share arguments with each other. in the figure, the predicates are drawn in the form of a network, based on the variables that they share. the meaning representation of a question can naturally be represented as a query, i.e. a series of steps that need to be taken in order to answer the question. each predicate represents a step in this reasoning process, and intuitively corresponds to an atomic cognitive operation that a human or machine can perform. in the case of the example utterance 'how many rubber spheres are there?', the reasoning process consists of four main steps. the first predicate, get-context, binds the image to the variable '?source'. then, the filter predicate filters the image for instantiations of the concept of sphere. the result of this filtering operation, i.e. the set of all spheres that are in the image, is bound to the variable '?spheres'. this set of spheres is subsequently filtered by another filter predicate for instantiations of the concept of rubber. the resulting set of rubber spheres is bound to the variable '?rubber-spheres'. finally, the set of rubber spheres is counted by the count predicate and the result is bound to the variable '?nr-of-rubber-spheres'. the meaning of the question 'how many rubber spheres are there?' corresponds thus informally to filtering an image for spheres, filtering the spheres for rubber objects and counting the result of this how many rubber spheres are there? (get-context ?source) (filter ?spheres ?source ?sphere) (bind shape-category ?sphere sphere) (bind material-category ?rubber rubber) (filter ?rubber-spheres ?spheres ?rubber) (count ?nr-of-rubber-spheres ?rubber-spheres) last operation. such meaning representations are called procedural semantic representations as the representations themselves are at the same time executable procedures @xcite . our methodology handles procedural semantic representations without problems, but is in no way restricted to it. it can handle any semantic representation, as long as it embraces some notion of compositionality and can be expressed as a set of predicates. examples of other compatible semantic representations include abstract meaning representation @xcite , propbank frames @xcite and the lambda calculus @xcite . the clevr dataset consists of three splits: a training split of 70,000 images and 699,989 questions, a validation split of 15,000 images and 149,991 questions, and a test split of 15,000 images and 149,988 questions. the questions in the training and validation splits come with semantic annotations, whereas the test set does not. as we require these annotations in order to evaluate our model, we use the training split of the clevr dataset as training set and the validation split as test set. the question-annotation pairs embrace various aspects of reasoning, including attribute identification ('there is a large cube; what is its color?'), counting ('how many green spheres are there?'), comparison ('are there an equal number of large cubes and small things?'), spatial relationships ('what size is the cylinder that is right of the yellow shiny thing that is left of the cube?') and logical operations ('how many objects are either red cubes or yellow cylinders?'). for the purposes of this paper, we have selected the subset of clevr questions that do not involve comparison, spatial relationships or logical operations. the main reason for this is that these are complex cognitive operations that often correspond to long and complex utterances that are far removed from the linguistic expressions that children (or even other humans) are faced with. our final training and test sets consist of 47,134 questions and 10,044 questions respectively. the learning task that we address consists in operationalising pattern finding mechanisms that facilitate the learning of a bidirectional construction grammar. the grammar should be able to map between the clevr utterances and their semantic representation, both in the comprehension (form to meaning representation) and the production (meaning representation to form) direction. ## methodology the input to the learning process consists of utterances that are annotated with a representation of their meaning. the output of the learning process should consist in form-meaning mappings (constructions) that can be used for comprehending and producing utterances. the form-meaning mappings are represented in, and processed using, fluid construction grammar @xcite @xcite . ## experiments this section presents a validation of our methodology for acquiring constructions on the clevr dataset discussed in section 2. we first describe the experimental set-up (section 4.1) and then present the evaluation results (section 4.2). ## related work prior mechanistic models that operationalise the learning of constructions can be divided into two groups, based on the learning task that they address. a first class of models learns constructions paired with their meaning representation, either provided in the form of an annotated corpus @xcite @xcite or obtained through task-oriented communicative interactions in a tutor-learner scenario @xcite @xcite . a second class of models, as introduced by @xcite , is designed to learn form-meaning pairings under referential uncertainty. as such, the exact meaning representations of the input utterances are not provided to the learning algorithm, but grammars are learnt based on the combination of input utterances and situational context snippets. in these experiments, input utterances always correspond to a single term in the situational context. in general, both classes of models have explored interesting ideas on a rather small scale, either because they were limited to specific linguistic phenomena @xcite @xcite @xcite @xcite @xcite , or because of the limited morpho-syntactic and semantic complexity of the input utterances @xcite @xcite @xcite @xcite . in all of the aforementioned work, either a segmentation of the input utterances, a lexicon or a set of predefined grammatical categories was provided. with the exception of the studies by gaspers et al., the corpora that were used to learn and evaluate the models were not made available and were not described in sufficient detail to make reproduction and comparison feasible. ## discussion and conclusion the scientific contribution of the methodology and experiments presented in this paper is twofold. on the one hand, they provide computational evidence for the cognitive plausibility of constructivist theories of language acquisition. these theories, as most prominently put forward by @xcite , attribute the ability of children to acquire language to two main cognitive capacities: intention reading and pattern finding. intention reading deals with reconstructing the intended meaning of observed utterances, while pattern finding implements generalisation processes that distil these reconstructed utterance-meaning pairs into abstract schemata embodying the linguistic knowledge of a language user. these schemata can then be used to fulfil the communicative function of language through the comprehension and production of natural language expressions. the methodology introduced in this paper presents a mechanistic model of the pattern finding capacity. based on utterances paired with a representation of their meaning, the learning algorithm gradually builds up an inventory of concrete to abstract form-meaning mappings, called constructions, along with a network of emergent grammatical categories that captures how the constructions of the grammar can combine to collaboratively comprehend and produce utterances. the experiments show that a small number of general learning operators, which become active if an utterance cannot be successfully processed by the grammar learnt so far, effectively leads to learning dynamics that are similar to those described in the psycholinguistic literature @xcite @xcite . in the first phase of the learning process, the learner acquires holistic mappings between utterances and their meaning representation. soon after that, holophrase constructions are generalised to item-based constructions that integrate a variable slot. at the same time, this generalisation process leads to the emergence of slot-filling constructions, here called lexical constructions. along with the item-based and lexical constructions, a network of grammatical categories emerges, capturing the distribution of construction slots and their observed fillers. in a third phase, more abstract itembased constructions emerge, with an increasingly large number of variable slots. in the final phase of the learning process, most constructions have already been acquired and most remaining impasses can be solved by adding new links to the categorial network. the learning dynamics are influenced by the degree of entrenchment of constructions. constructions that are often successfully used become more entrenched, while their competitors are suppressed. as a result of this entrenchment process, the grammar reaches a stable state, while it remains adaptive to any changes in the discourse or environment. similar dynamics have been observed in earlier experiments in the field of evolutionary linguistics, for instance in experiments on the emergence of compositionality in a population of autonomous agents @xcite . on the other hand, the methodology and experiments presented in this paper pave the way for learning computationally tractable, large-scale, usage-based grammars that facilitate both language comprehension and production. the proposed learning algorithm supports online, interactive, incremental, transparent and data-efficient learning. the learner builds up its human-interpretable inventory of constructions and categories through the application of transparent syntactico-semantic generalisation processes. already after a single observation, the fragment of linguistic knowledge acquired by the learner can be successfully used for language comprehension and production. as more and more utterance-meaning pairs are observed, the linguistic knowledge of the learner quickly expands and becomes better fit for achieving their communication goals. as a result of the dynamics of rewarding successful construction applications and punishing competing ones, the grammar of the learner remains ever-adaptive to any changes in the task or environment. due to their online, interactive, incremental, transparent and data-efficient nature, we strongly believe that the proposed mechanisms for learning computational construction grammars can serve as an excellent basis for implementing the language acquisition ability of truly intelligent agents.