Learning to Understand Child-directed and Adult-directed Speech (2020)

Abstract: Speech directed to children differs from adult-directed speech in linguistic aspects such as repetition, word choice, and sentence length, as well as in aspects of the speech signal itself, such as prosodic and phonemic variation. Human language acquisition research indicates that child-directed speech helps language learners. This study explores the effect of child-directed speech when learning to extract semantic information from speech directly. We compare the task performance of models trained on adult-directed speech (ADS) and child-directed speech (CDS). We find indications that CDS helps in the initial stages of learning, but eventually, models trained on ADS reach comparable task performance, and generalize better. The results suggest that this is at least partially due to linguistic rather than acoustic properties of the two registers, as we see the same pattern when looking at models trained on acoustically comparable synthetic speech.

## introduction speech directed to children (cds) differs from adult-directed speech (ads) in many aspects. linguistic differences include the number of words per utterance, with utterances in cds being considerably shorter than utterances in ads, and repetition, which is more common in child-directed speech. there are also paralinguistic, acoustic factors that characterize child-directed speech: people speaking to children typically use a higher pitch and exaggerated intonation. it has been argued that the properties of cds help perception or comprehension. @xcite propose that cds is optimized for learnability. optimal learnability may, but does not necessarily align with optimization for perception or comprehension. although speech with lower variability may be easiest to learn to understand, higher variability may provide more learning opportunities, leading to more complete language knowledge. in this paper, we explore how learning to extract meaning from speech differs when learning from cds and ads. we discuss task performance on the training register as well as generalization across registers. to tease apart the effect of acoustic and linguistic differences, we also report on models trained on synthesized speech, in which linguistic differences between the registers are retained, but the acoustic properties are similar. the characteristics of child-directed speech are a major topic of study in language acquisition research. for a comprehensive overview, see @xcite and @xcite . with regards to acoustics, cds is reported to have exaggerated intonation and a slower speech rate @xcite . @xcite show that cds contains more 'extreme' realizations of vowels. @xcite show that these increased means are accompanied by increased variance, and argue that any learning advantage of cds due to extreme vowel realizations is counteracted by increased variance. however, it has also been argued that increased variance may be beneficial to learning in the long run, as it gives the learner a more complete set of examples for a category, which helps generalization. @xcite show that word forms in childdirected speech are acoustically more diverse. at the utterance level, child-directed language consists of shorter sentences and simpler syntax @xcite , and words more often appear in isolation @xcite . studies on home recordings show that the availability of cds input accounts for differences in vocabulary growth between learners, whereas overheard speech is unrelated @xcite . this does not necessarily mean that it is easier to learn from cds. psycholinguistic research has shown that infants across the world show a cds preference, paying more attention to it than to ads (manybabies consortium, 2020). learning advantages of cds in children may therefore simply be because they grant it more attention, rather than to properties of cds that are advantageous for learning. computational models, however, have no choice in where they allocate attention. any learning advantages we find of either ads or cds in computational studies must be due to properties that make speech in that register more learnable to the model. there has been some computational work comparing learning from ads and cds at the level of word learning and phonetic learning. studies on segmentability use algorithms that learn to identify word units, with some studies reporting higher segmentability for cds @xcite , while @xcite report mixed results. @xcite train hmm-based speech recognition systems on cds and ads, and test on matched and crossed test sets. they find that both ads and cds trained systems perform best on the matching test set, but cds trained systems perform better on ads than systems trained on ads peform on cds. they show that this is likely caused by phonetic classes have larger overlaps in cds. to the authors' knowledge, the current work is the first to computationally explore learnability differences between ads and cds considering the process of speech comprehension as a whole: from audio to semantic information. ## task the task is to match speech to a semantic representation of the language it contains, intuitively 'grounding' it to the semantic context. the design of this task is inspired by work in visual grounding. however, the availability of cds data accompanied by visual data is very limited. instead of visual representation, we use semantic sentence embeddings of the transcriptions. rather than training our model to imagine the visual context accompanying an utterance, as in visual grounding, we train it to imagine the semantic content. note that since the semantic embeddings are based on the transcriptions of the sentences themselves, they have a much closer relation to the sentences than visual context representations would have. the semantic sentence representations were obtained using sbert, a bert-based architecture that yields sentence embeddings, which was finetuned on the sts benchmark of semeval (reimers and gurevych, 2019). this particular encoding was chosen because it harnesses the semantic strength of bert @xcite in an encoding of the sentence as a whole. speech is converted melfrequency cepstrum coefficients. child-directed speech in the newmanratner corpus takes place in free play between caregiver and child, whereas adult-directed speech is uttered in the context of an interview. stretches of speech have been transcribed containing one or more utterances. we selected only utterances by caregivers and excluded segments with multiple speakers. as the cds portion of the corpus is larger than the ads portion, we randomly selected 21,465 cds segments, matching the number of ads segments by caregivers. validation and test sets of 1,000 segments were held out, while the remaining 19,465 segments were used for training. ## model the model and training set-up is based on @xcite . this model is suited to our task, as it allows to learn to extract semantic information from speech by grounding it in another modality, without requiring the speech to be segmented. the speech encoder comprises a convolutional filter over the speech input, feeding into a stack of 4 bidirectional-gru layers followed by an attention operator. the difference in our set-up is the use of sbert sentence embeddings instead of visual feature vectors. using a margin loss, the model is trained to make the cosine distance between true pairs of speech segments and sbert embeddings smaller than that between random counterparts. we train for 50 epochs and following @xcite we use a cyclic learning rate schedule. foot_0 6 results ## discussion we find indications that learning to extract meaning from speech is initially faster when learning from child-directed speech, but learning from adultdirected speech eventually leads to similar task performance on the training register, and better generalization to the other register. the effect is present both in models trained on natural speech and in models trained on synthetic speech, suggesting that it is at least partly due to differences in the language itself, rather than acoustic properties of the speech register. our finding that models trained on ads generalize better to cds than the other way around contrasts with the findings of @xcite . our results are in contrast to the idea that cds is optimized for leading to the most valuable knowledge, as it is the models trained on ads that lead to better generalization. our finding that learning is initially faster for cds is more in line with the idea of learnability as 'easy to learn'. the better generalization of models trained on ads may be due to ads having higher lexical and semantic variability, reflected in the larger vocabulary and higher number of words per utterance. since there is simply more to learn, learning to perform the task is more difficult on ads, but it leads to more valuable knowledge. it is also possible that sbert is better suited to encode the semantic content of ads, as ads uterrances are likely to be more similar to the sentences sbert was trained on than cds utterances are. we must be prudent in drawing conclusions from the apparent effects we see in this study, as the results on different datasets cannot be interpreted as being on the same scale. although all metrics are based on a rank of the same number of competitors, the distribution of similarities and differences between the semantic representations of these competitors may differ across datasets. the combined test set scores are more directly comparable, but ideally, we would like to compare the generalization of both models on an independent test set. in future work, we intend to curate a test set with data from separate sources, which can serve as a benchmark for the models we study. we intend to explore how a curriculum of cds followed by ads affects learning trajectories and outcomes. we also intend to use tools for interpreting the knowledge encoded in neural networks (such as diagnostic classifiers and representational similarity analysis) to investigate the emergent representation of linguistic units such as phonemes and words.