Modeling the Interaction Between Perception-Based and Production-Based Learning in Children’s Early Acquisition of Semantic Knowledge (2021)

Abstract: Children learn the meaning of words and sentences in their native language at an impressive speed and from highly ambiguous input. To account for this learning, previous computational modeling has focused mainly on the study of perception-based mechanisms like cross-situational learning. However, children do not learn only by exposure to the input. As soon as they start to talk, they practice their knowledge in social interactions and they receive feedback from their caregivers. In this work, we propose a model integrating both perception- and production-based learning using artificial neural networks which we train on a large corpus of crowd-sourced images with corresponding descriptions. We found that production-based learning improves performance above and beyond perception-based learning across a wide range of semantic tasks including both word- and sentence-level semantics. In addition, we documented a synergy between these two mechanisms, where their alternation allows the model to converge on more balanced semantic knowledge. The broader impact of this work is to highlight the importance of modeling language learning in the context of social interactions where children are not only understood as passively absorbing the input, but also as actively participating in the construction of their linguistic knowledge.

## introduction an important aspect of language acquisition is learning how to map linguistic forms to meanings. this involves both mapping individual word forms (e.g., "dog") to concepts of the world (e.g. the category dog) and mapping the relationship between words in a sentence (e.g., "the dog chases the ball") to a given event configuration in the world (i.e., that the dog is the agent performing the act of chasing on the ball, the semantic patient). children manage to learn this mapping in their native language at an impressive speed @xcite @xcite and despite the high ambiguity of this task in the natural context where language learning occurs @xcite . ## discussion how do children learn the meanings of words and sentences in their native language? previous modeling effort has largely focused on perception-based learning mechanisms such as xsl. however, children do not learn only by mere exposure to the perceptual cross-modal input, they also practice their early -albeit rudimentary-knowledge and receive feedback from caregivers, which allows them to correct/refine this knowledge @xcite , 2020). here we investigated one possible feedback mechanism on children's early production (cf), that relies on general coordination and miscoordination cues, and does not necessarily require the caregiver providing an explicit correction. we proposed a computational model that integrates both xsl and cf, allowing us to study how these two mechanisms could interact in early semantic learning. the same model learns both from perceptual input and from feedback on production through reinforcement. we tested various learning scenarios that varied in their plausibility given our understanding of how children's learn language. crucially, we found that the most plausible learning scenario (i.e., xsl+alt) -where the model first learns through perception, and second through alternating perception and production -is also the one that leads to the best overall performance on most semantic tasks. the fact that xsl+alt performed better than xsl alone confirms the main hypothesis of this work: cf plays a role in semantic learning above and beyond xsl. in addition, the fact that altwhich alternates perception and production from the start -hurts performance compared to xsl, suggests that for cf to be effective, it requires a first phase of learning through perception, which is an intuitive finding since the model has first to be exposed to enough linguistic/semantic input to be able to start producing -at least partially -meaningful utterances (for which rl is more useful). this finding also corresponds to children's learning trajectory where they only start producing words (and receiving feedback on them) after a period of pure perception-based learning. 10 interactions between perception-based and production-based learning another interesting finding of this work is that xsl+alt (e.g., alternating xsl and cf after a period of pure xsl) performs better than xsl+cf (i.e., using cf alone after a period of pure xsl). this finding means that when cf is combined with xsl, it leads to improvement in performance compared to when either xsl or cf operates alone or in a sequential fashion. in other words, we found that xsl and cf interact synergistically to improve performance. in what follows, we examine this observed synergy in more details. results in qualitative and quantitative investigation of the model's behavior supports our speculation. for example, when we sample sentences randomly from the productions of xsl+cf and xsl+alt given images in the validation set, we observed that while xsl+cf produces a variety of verbs (similar to xsl+alt), it tends to produce systematically shorter utterances involving disproportionately only one person (see figure 4 confirms this observation quantitatively: xsl+cf increasingly produces sentences involving jenny, but decreasingly sentences involving mike. this fact leads to the situation where the model gets less feedback on the difference between jenny and mike and, therefore, unlearns this distinction to some degree. for xsl+alt, the fraction of sentences involving jenny and mike remains largely constant, thus avoiding the problem faced by xsl+cf. at the son and swingley, 2012), indicating that they engage in a perception-based learning well before starting to produce their own utterances. 11 the drop in "persons" could explain the slight drop in "semantic roles," (as distinguishing the persons is a prerequisite to understand semantic roles) however this slight drop is within the margin of error, so we could not draw strong conclusions about the difference with xsl for this category. same time, xsl+alt keeps a balanced coverage of verbs allowing it to maintain the good scores achieved by xsl+cf on this category (see appendix f for a quantitative analysis comparing the production of verbs in both models). the conclusion we draw from comparing xsl+cf and xsl+alt is that, even after a period of pure xsl, continuing to learn through xsl from time to time while doing reinforcement on production helps the model not to get biased towards a subset of the language it is supposed to learn. similar phenomena of "language drift" -due to reinforcement learning operating alone -have been observed in another line of work studying emergent communication systems @xcite @xcite . learning trajectories the best performing model, i.e. xsl+alt, not only instantiates -intuitively -the most plausible learning scenario in early childhood, it also recapitulates some specific findings in the language development literature about the timeline of semantic learning. for example, it learns nouns before predicates (adjectives and verbs), resonating with previous findings about the "noun bias" @xcite @xcite . that said, the models' performance on verbs (relative to other parts of speech) should be interpreted with caution given the fact that we only used static images in both training and testing. in real life, children learn verbs from dynamic actions and some experimental studies also evaluate verb learning use videos instead of static images @xcite . the model shows a rather late onset of understanding sentence-level semantics such as semantic roles, only after a sizable lexicon has been acquired. this fact mirrors, e.g., the finding that children show evidence of recognize semantic roles in a sentence during their second year of life @xcite , that is, at an age when they have already acquired a substantial vocabulary @xcite . note that the model's performance on sentence-level semantics remains relatively low compared to word-level semantics even when learning from the entire dataset. it is difficult, based only on the current results, to conclude whether more data will lead to improvement in sentence-level semantics or whether the model has already reached its ceiling performance due to structural limitations (e.g., the lack of higher-level conceptual knowledge about semantic agency).