Evaluating the Acquisition of Semantic Knowledge from Cross-situational Learning in Artificial Neural Networks (2021)

Abstract: When learning their native language, children acquire the meanings of words and sentences from highly ambiguous input without much explicit supervision. One possible learning mechanism is cross-situational learning, which has been successfully tested in laboratory experiments with children. Here we use Artificial Neural Networks to test if this mechanism scales up to more natural language and visual scenes using a large dataset of crowd-sourced images with corresponding descriptions. We evaluate learning using a series of tasks inspired by methods commonly used in laboratory studies of language acquisition. We show that the model acquires rich semantic knowledge both at the word- and sentence-level, mirroring the patterns and trajectory of learning in early childhood. Our work highlights the usefulness of low-level co-occurrence statistics across modalities in facilitating the early acquisition of higher-level semantic knowledge.

## introduction in order to acquire their native language, children learn both how to associate individual words with their meanings (e.g., the word "ball" refers to the object ball and the word "kick" refers to that act of kicking) and how to map the relationship between words in a sentence onto specific event configurations in the world, e.g., that the sequence of words "jenny kicks the ball" maps on to the event where the referent of the first noun (i.e., jenny) is performing the act of kicking on the second (i.e., the ball). this is a difficult task because it requires that children learn these associations and rules in a largely unsupervised fashion from an input that can be highly ambiguous @xcite . it is still unclear how children overcome this challenge. previous experimental studies on child language acquisition have focused on evaluating chil-dren's learning using controlled tasks that typically take the form of a two-alternative forced-choice paradigm. for example, in order to test the learning of an individual word meaning, we can utter this word to the child (e.g., "ball") and present her with two pictures representing correct (i.e., a ball) and incorrect referents (e.g. a cup), and we test if the child reliably prefers the correct one @xcite . similarly, in order to evaluate children's understanding of sentence-level semantics such as a the agent-patient relationship, we can utter a sentence such as "jenny is tickling mike" and present the child with two pictures where either jenny or mike are doing the tickling, and we test if the child reliably prefers the correct picture (e.g. @xcite . while we have been able to evaluate children's knowledge using such controlled tests, research has been less compelling regarding the mechanism of learning from the natural, ambiguous input. one promising proposal is that of cross-situational learning (hereafter, xsl). this proposal suggests that, even if one naming situation is highly ambiguous, being exposed to many situations allows the learner to narrow down, over time, the set of possible wordworld associations (e.g. @xcite . while in-lab work has shown that xsl is cognitively plausible using toy situations @xcite , effort is still ongoing to test if this mechanism scales up to more natural learning contexts using machine learning tools (e.g. @xcite . this previous work, however, has focused mainly on testing the learning of individual words' meanings, while here we are interested in testing and comparing both word-level and sentence-level semantics. ## results to evaluate the learned semantic knowledge, we measure, for each task, the model's accuracy at rating the similarity of the image and the target sentence γ(i, s t ) higher than the similarity to the distractor sentence γ(i, s d ). we report both final accuracy scores after the model has converged as well as intermediate scores before convergence, which we take as a proxy for the learning trajectory. to ensure reproducibility, we make the semantic evaluation sets as well as the source code for all experiments publicly available. foot_8 ## discussion this paper dealt with the question of how children learn the word-world mapping in their native language. as a possible learning mechanism, we investigated xsl, that has received much attention in the literature. while laboratory studies on xsl have typically used very simplified learning situations to test if children are cognitively equipped to learn a toy language in an xsl fashion. the question remains as whether such a mechanism scales up to the learning of real languages where the learning situations can be highly ambiguous. the novelty of our work is that we were interested not only in the scalability of xsl to learn from more naturalistic input, but also its scalability to the learning of various aspects of semantic knowledge. these include both the meanings of individual words (belonging to various categories such as nouns, adjectives, and verbs) and the meanings of higher level semantics such as the ability to map how words relate to each other in the sentence (e.g., subject vs. object) to the semantic roles of their respective referent in the world (e.g., agent vs. patient/recipient). we were able to perform these evaluations using a simple method inspired from the field of experimental child development and which has usually been used to test the same learning phenomena in children, i.e., the two-alternative forced choice task. using this evaluation method, we found that an xsl-based model trained on a large set of pictures and their descriptions was able to learn word-level meanings for nouns and verbs relatively well, but struggles with adjectives. further, the model seems to learn some sentence-level semantics, especially verb-noun dependencies and semantic roles. finally, concerning the learning trajectory, the model initially learns the semantics of nouns and only later the semantics of verbs and more complex sentence-level semantics. concerning word-level semantics, the fact that the model learns nouns better than (and before) the predicates (adjectives and verbs) resonates with findings in child development about the "noun bias" @xcite @xcite . the model also learns verbs better than adjectives. however, we suspect this finding is caused by the limited availability of adjectives in the dataset. 12 in fact, the verb-related actions (e.g. "sitting" vs. "standing") were arguably more salient and easier to detect visually than adjective-related words ("happy" vs. "sad") which require a finegrained detection of the facial expressions. concerning sentence-level semantics, the model performed surprisingly well on verb-noun dependency task where the model assigned a semantic role to one participant and on the similar but (arguably) more challenging task of assigning semantic roles to two participants. further, the fact that the model shows a rather late onset of understanding of semantic roles, only after a set of nouns and verbs have been acquired (cf. figure 3 ) mirrors children's developmental timeline. indeed, children become able to assign semantic roles to nouns in a sentence correctly when they are around 2 years and 3 months old @xcite , at an age when they have already acquired a substantial vocabulary including many lexical categories such as nouns and verbs @xcite in this paper, we used artificial neural networks to study how properties the input can (ideally) inform the learning of semantics. our modeling did not purport to account for the details of the cognitive processes that operate in children's minds nor did it take into account limitations in children's information-processing abilities. thus, this work is best situated at the computational level of analysis @xcite , which is only a first step towards a deeper understanding of the precise algorithmic implementation. that said, we can speculate about the internal mechanisms used by the model to succeed in the tasks and about their potential insights into children's own learning. for example, it is very likely that the model leverages simple heuristics to recognize the agent in a sentence, e.g., it may have learned to associate the first appearing noun in the sentence to the agent of the action. research on child language suggest that children also use such heuristics (e.g. @xcite . this suggests that the model, like children, might use partial representations of sentence structure (i.e., rudimentary syntax) to guide semantic interpretation. exploiting structural properties of the input (e.g., order of words in a sentence) may be insightful when it mirrors genuine learning heuristics in children. however, a neural network model may also capitalize on idiosyncratic biases in the dataset (that do not reflect the natural distribution in the world) to achieve misleadingly high performance. 13 for example, a misleading bias in the linguistic input is if a certain noun (e.g., jenny) occurs more fre-ases such as the agent spatial order in the images. indeed, investigation of our semantic roles test set shows that the agent occurs roughly equally on the right (52%) and left sides, which means that a model exploiting such a bias could only perform around chance level. there could be other biases we are not aware of and which require performing further controls. that said, this is an open question for all research using neural networks as models of human learning. more generally, our understanding of language acquisition would greatly benefit from further research on the interpretation of neural network learning, revealing the content of these black box models. this would allow us to tease apart genuine insights about realistic heuristics that could be used by children and artificial shortcuts that only reflect biases in the learning datasets. in future work, we plan to study visual datasets with even more naturalistic scenes such as coco @xcite . in this regard, maybe closer to our work is the study by @xcite who used coco to create a set of distractor captions to analyze whether vision and language models are sensitive to (maximally difficult) singleword replacements. our goal is to go beyond these analysis to test specific semantic phenomena as we did here with the abstract scenes dataset. another step towards more naturalistic input is the use speech input instead of text @xcite . finally, this work focused on testing how xsl scales up to natural language learning across many semantic tasks. nevertheless, children's language learning involves more than the mere tracking of cooccurrence statistics: they are also social beings, they actively interact with more knowledgeable people around them and are able to learn from such interactions @xcite . future modeling work should seek to integrate both statistical and social learning skills for a better understanding of early language learning.