Language Acquisition through Intention Reading and Pattern Finding (2022)

Abstract: One of AI’s grand challenges consists in the development of autonomous agents with communication systems offering the robustness, flexibility and adaptivity found in human languages. While the processes through which children acquire language are by now relatively well understood, a faithful computational operationalisation of the underlying mechanisms is still lacking. Two main cognitive processes are involved in child language acquisition. First, children need to reconstruct the intended meaning of observed utterances, a process called intention reading. Then, they can gradually abstract away from concrete utterances in a process called pattern finding and acquire productive schemata that generalise over form and meaning. In this paper, we introduce a mechanistic model of the intention reading process and its integration with pattern finding capacities. Concretely, we present an agent-based simulation in which an agent learns a grammar that enables them to ask and answer questions about a scene. This involves the reconstruction of queries that correspond to observed questions based on the answer and scene alone, and the generalization of linguistic schemata based on these reconstructed question-query pairs. The result is a productive grammar which can be used to map between natural language questions and queries without ever having observed the queries.

## introduction language is a unique hallmark of human intelligence. our linguistic systems do not only meticulously serve our communicative needs, they are also incredibly robust to noise, adaptive to change and they can be learnt efficiently. while the processes that drive language acquisition in children are by now relatively well understood, a faithful computational operationalisation of the underlying mechanisms is still lacking. having such a mechanistic model would, however, constitute a crucial * shared last authors. the idea that children acquire language by actively participating in communicative interactions and making use of general cognitive capacities has been elaborately documented in studies on usage-based language acquisition @xcite @xcite . in particular, two highly complementary cognitive processes have been identified to play a key role: intention reading and pattern finding @xcite . first, children need to understand the communicative intentions of their interlocutors. in a process called intention reading, they reconstruct the intended meaning of the utterances they observe. then, they can gradually abstract away from concrete utterances and meaning representations in a process called pattern finding, and acquire productive schemata that generalise over form and meaning. theoretical as well as empirical evidence has been abundantly provided for both intention reading @xcite @xcite and pattern finding @xcite @xcite in this paper, we introduce a mechanistic model of the intention reading process and its integration with pattern finding capacities. concretely, we present an agent-based simulation in which an artificial agent learns a construction grammar that enables it to ask and answer questions about scenes it observes. the learning task thus involves the reconstruction of queries that correspond to observed questions based on the answer and scene alone, as well as the generalization of linguistic schemata based on these reconstructed question-query pairs. the learner gradually acquires a fully productive grammar, consisting of form-meaning mappings, that can be used for both language comprehension, i.e. observing a question and mapping it onto a query, and language production, i.e. expressing a query in the form of an interrogative linguistic expression. when it comes to intention reading, the learning challenge amounts to the reconstruction of a query based on a question-answer pair and a scene, without ever observing the query itself. the learner agent is endowed with an inventory of primitive operations, which it can combine to compose new queries. the query composition process allows the agent to hypothesize about the meaning of a question given the scene and the answer to the question. the space of all possible queries that lead to the observed answer in the given scene is typically very large. at the same time, most of these queries are not adequate representations of the meaning of the question and only lead to the correct answer in this specific scene. the second challenge is to learn abstract schemata. pairing an observed utterance with its reconstructed meaning yields a form-meaning mapping, called a construction @xcite . initially, the learner has no way of knowing which parts of the form correspond to which parts of the meaning. hence, it stores this mapping holistically. through the observation of different form-meaning mappings, pattern finding mechanisms allow the agent to generalise over reoccurring form-meaning patterns, thereby capturing the compositional structure of the language. intention reading and pattern finding are highly complementary. specifically, intention reading facilitates pattern finding by providing meaning hypotheses. in turn, pattern finding constrains the search process involved in intention reading by providing partial analyses. this interplay between intention reading and pattern finding is key in successfully tackling the learning challenge, and constitutes the main contribution of this paper. we validate our methodology using the clevr benchmark dataset @xcite , in which the communicative task of the agents consists in asking and answering questions about scenes of geometric figures. over many interactions, the learner incrementally acquires a fully operational grammar that can be used for both language comprehension and production. we show that the acquired grammar effectively solves the visual question answering task. the contributions of this work are both theoretical and practical. on the theoretical side, the presented work provides computational evidence for the cognitive plausibility of usage-based theories of language acquisition, in particular concerning intention reading and pattern finding. on the practical side, this paper introduces a powerful new methodology that allows autonomous agents to acquire an effective communication system with human-like properties through task-oriented interactions in their native environment. the remainder of this paper is structured as follows. section 2 introduces the dataset. in section 3, we introduce the technical foundations of our methodology. section 4 describes the experimental setup for learning construction grammars through communicative interactions. section 5 presents the experimental results. related work is discussed in section 6. finally, section 7 reflects on the results and contributions of our work. the code of this experiment is made available through the open-source babel toolkit foot_0 @xcite . ## data the clevr dataset @xcite consists of (i) rendered scenes with geometric figures of various shapes, sizes, colours and materials, (ii) english questions about these scenes, and (iii) answers to these questions. the questions test a variety of reasoning skills, including attribute identification ("what size is the yellow cube?"), counting ("how many large cylinders are there?"), existence ("is there a red ball?"), comparison ("are there more spheres than cylinders?"), spatial relationships ("what shape is the thing right of the purple cube?") and logical operations ("how many things are either spheres or cylinders?"). this dataset was chosen because it satisfies two criteria. first, it offers visually grounded linguistic expressions. the objects about which the agents communicate are actual referents in the agents' environment. second, it contains a large number of scenes and plenty of similar, yet non-identical questions. such examples are necessary for any kind of generalisation process and are consistent with theoretical and empirical evidence of how children learn language @xcite . other datasets that fit these two criteria could also be used. for the experiment in this paper, we have selected a subset of the clevr questions. specifically, questions concerning comparison, spatial relations and logical operations have been left out. the main reason for this is that these are more complex cognitive operations that correspond to longer and more complex questions. such questions are far removed from the type that children are faced with. starting from clevr's validation split, our final dataset consists of 10,044 unique questions. each question can be used in any of the 15,000 scenes of the validation split. ## technical foundations our methodology builds on three main technical foundations: (i) the language game paradigm (section 3.1), (ii) procedural cognitive semantics (section 3.2) and (iii) computational construction grammar (section 3.3). ## the elicitation game we set up a language game in a tutor-learner scenario, which we call the elicitation game. the agents are situated in scenes from the clevr dataset. the tutor is an agent that has an established linguistic inventory which allows to comprehend and produce all questions from the clevr in the following sections, the processes of conceptualisation, production, comprehension and interpretation are discussed in detail. afterwards, the learning mechanisms operationalising intention reading and pattern finding are introduced. ## experimental results figure 5 presents the main results of the experiment. it shows the communicative success and the grammar size over time. both metrics start at 0. the communicative success rises quickly, with more than 78% of the interactions being successful after 5,000 games. this is half the number of possible utterances from our subset of data. the grammar size also reaches its peak at this point, with on average 1,048 constructions being learned. success reaches over 98% after 10,000 interactions and does not go below 99.9% from interaction 25,000 onwards. it is only after 200,000 interactions that the success reaches a s figure 6 breaks down the grammar size per con-struction type. at the start of the experiment, only holophrase constructions are learned. soon after, the learner can start to generalise over them. the construction inventory peaks after 5,000 interactions, reaching 184 holophrase constructions and 838 item-based constructions. afterwards, more abstract item-based constructions gradually become dominant and overtake their less abstract competitors, including the holophrase constructions. by the end of the experiment, 31 holophrase constructions and 85 item-based constructions remain. there is less competition among the lexical constructions. after 10,000 interactions, 33 lexical constructions are learned and this remains stable until the end. the theoretical maximum of 35 lexical constructions was reached in four out of ten experimental runs. after 10,000 interactions, 33 lexical constructions are learned and this remains stable until the end. we note that it is not the goal of the experiment to reach one particular set of constructions. hence, the absolute number of constructions at a given time is not important. instead, the goal is to become successful at the communicative task and learn an efficient construction inventory for doing so. ## related work prior agent-based models have also studied the constructivist co-acquisition of syntax and semantics through task-oriented interactions @xcite @xcite . while these models have provided important insights, this paper advances the state of the art in two ways. first, the presented experiment operates on a much larger scale. in contrast to prior models, this work does not focus on a specific linguistic phenomenon, such as the russian aspectual system @xcite , the hungarian agreement system @xcite or english spatial language @xcite . the utterances being considered in this work are far more complex, both in terms of morpho-syntax and semantics. second, the presented experiment provides fewer scaffolds. the agent does not receive a segmentation of input utterances, a predefined lexicon as in @xcite or a taxonomy guiding the generalisation process of constructions as in @xcite . the agent only relies on a number of basic cognitive operations and a collection of concepts. the latter can also be learned from sub-symbolic observations through the language game methodology, as shown by @xcite . when it comes to the field of visual question answering, existing approaches typically tackle the task in one of two ways. either, a large end-to-end neural network maps an image and a question to the answer, e.g. as in @xcite . alternatively, rnns are used to map the question onto a query which is then executed on the image, e.g. as in @xcite . both of these approaches rely on huge amounts of training data. the second approach additionally requires questions annotated with queries to train the rnn. further, both approaches rely on black-box architectures, making it unclear how and why an answer was generated. our methodology overcomes these shortcomings. similar to the first approach, the agent is only presented with images, questions and their answers and autonomously reconstructs the underlying queries. our methodology is more dataefficient and the agents' representations and reasoning processes are fully transparent. finally, the agents' communication system is open-ended and completely bidirectional using the same representations and processing mechanisms, which is not possible using current neural network architectures. ## concluding discussion the contributions of this paper span two areas.