Learning E nglish with P eppa P ig (2022)

Abstract: Recent computational models of the acquisition of spoken language via grounding in perception exploit associations between spoken and visual modalities and learn to represent speech and visual data in a joint vector space. A major unresolved issue from the point of ecological validity is the training data, typically consisting of images or videos paired with spoken descriptions of what is depicted. Such a setup guarantees an unrealistically strong correlation between speech and the visual data. In the real world the coupling between the linguistic and the visual modality is loose, and often confounded by correlations with non-semantic aspects of the speech signal. Here we address this shortcoming by using a dataset based on the children’s cartoon Peppa Pig. We train a simple bi-modal architecture on the portion of the data consisting of dialog between characters, and evaluate on segments containing descriptive narrations. Despite the weak and confounded signal in this training data, our model succeeds at learning aspects of the visual semantics of spoken language.

## introduction attempts to model or simulate the acquisition of spoken language via grounding in the visual modality date to the beginning of this century @xcite but have gained momentum recently with the revival of neural networks (e.g., @xcite @xcite @xcite @xcite @xcite @xcite . current approaches work well enough from an applied point of view, but most are not generalizable to real-life situations that humans or adaptive artificial agents expe-rience. commonly used training data consist of images or videos paired with spoken descriptions of the scene depicted: however, the type of input that a language learner receives from the environment is much more challenging. firstly, speech is only loosely coupled with the visual modality in naturalistic settings @xcite . speakers often mention concepts that are not present in the immediate perceptual context, or talk about events that are remote in space and/or time (for example past experiences or future plans). secondly, in addition to correlations between the visual scenes and the meaning of spoken utterances, there are also correlations with non-semantic aspects of the speech signal, such as the voices of specific speakers, as well as with non-speech ambient sounds. although it is plausible that such non-semantic correlations can sometimes be useful to the learner in the general endeavor of making sense of the world, for the specific task of learning the semantics of linguistic units they are likely more often an obstacle, as they make it harder to zoom in on the meaning-bearing aspects of the audio signal. in the current study we make a first step towards simulating the acquisition of language via grounding in perception in a more naturalistic scenario. our main focus is on learning the meaning of linguistic expressions from spoken utterances grounded in video. we use the well-known children's cartoon peppa pig as a case study. compared to commonly used video datasets, this dataset has a number of interesting characteristics. the visual modality is very schematic, the language is simple in terms of vocabulary size and syntactic complexity, and analysis of its linguistic features suggests its suitability for beginner learners of english @xcite . crucially, however, most of the speech in the videos consists of naturalistic dialogs between the characters in which they do not only discuss the here and now, but also often use displaced language. 1 thus, the utterances are only loosely and noisily correlated to the scenes and actions depicted in the videos. this choice of data thus allows us to directly address the ecological limitations of the current approaches. in addition, the cartoon videos contain comments interjected by the narrator. we use these for evaluating the acquisition of meaning as they are more descriptive and less noisy and allow us to measure performance, while controlling for speaker characteristics. we implement a simple bi-modal architecture that learns spoken language embeddings from videos, and train it on the peppa pig dataset. our contributions are the following: we show that despite the challenges of our naturalistic training data, our model succeeds at learning associations between the form of spoken utterances and their visual semantics. moreover, even though the model rarely hears words in isolation, it captures aspects of the visual meaning of frequent nouns and verbs. our ablation studies suggest that temporal information contributes to video modeling (especially for longer segments), and that self-supervised pre-training followed by fine-tuning of the audio encoder is key to the best performance. ## related work early attempts at simulating grounded language learning focus on interactions between adults and 1 for example, when daddy pig explains that they need to clean up before mummy pig sees the mess that peppa and george made, or when talking about plans to visit friends. young children while playing with a set of objects from different categories @xcite @xcite . in a representative study from this series, @xcite use speech recorded from such interactions paired with different views of the visible objects to identify linguistic units (i.e., words) and visual categories, and to map these two modalities together. a hard-coded visual system extracts object representations from images, and spoken utterances are represented as phoneme probabilities generated by an rnn pre-trained on spectrograms. their experiments on small-scale data (around 20 words and seven visual categories) show that the model can segment words and map them to visual categories. ## method the main focus of this study is on the data and evaluation. we thus keep the components of our architecture simple, and follow established modeling practices whenever possible. ## experimental settings we implement the architecture in pytorch @xcite . we use the adam optimizer (kingma and ba, 2015) with the scheduling described in @xcite . we train every configuration on a single gpu and stop training after 48 hours, with batch-size 8 and accumulating gradients over 8 batches, in 16 bit precision mode. for each model configuration we save model weights after each epoch and report results for the checkpoint which gets the best triplet accuracy on the narration validation data. our code is publicly available at https:// github.com/gchrupala/peppa , and can be consulted for further details of the experimental setup. ## results figure 3 shows recall@n for values of n between 1 and 10 for the complete model on test narration data in both fixed and jitter conditions. both plots show that the value of recall@n increases monotnically. for the rest of this paper, we only report recall@10. ## conclusion we simulate grounded language learning in a naturalistic setting, where the connection between the linguistic and visual modalities is not always strong and is potentially confounded by correlations with non-semantic aspects of the speech signal. our experimental results suggest that despite the challenges inherent to the naturalistic aspects of our training dataset, a simple bimodal architecture can capture aspects of visual meaning of individual words as well as full utterances, and generalize well to narrative utterances featuring a single unseen speaker and a descriptive rather than conversational style. our analyses show that generalization is substantially boosted by fine-tuning audio representations pretrained on unlabeled single-modality speech data. fine-tuning a pretrained video encoder also makes a contribution, but is less crucial to generalization from dialog to narration. we also investigate the role of temporal information in learning form-meaning mappings and show that having access to time information facilitates learning, except for very short video segments.