Acquiring language from speech by learning to remember and predict (2020)

Abstract: Classical accounts of child language learning invoke memory limits as a pressure to discover sparse, language-like representations of speech, while more recent proposals stress the importance of prediction for language learning. In this study, we propose a broad-coverage unsupervised neural network model to test memory and prediction as sources of signal by which children might acquire language directly from the perceptual stream. Our model embodies several likely properties of real-time human cognition: it is strictly incremental, it encodes speech into hierarchically organized labeled segments, it allows interactive top-down and bottom-up information flow, it attempts to model its own sequence of latent representations, and its objective function only recruits local signals that are plausibly supported by human working memory capacity. We show that much phonemic structure is learnable from unlabeled speech on the basis of these local signals. We further show that remembering the past and predicting the future both contribute to the linguistic content of acquired representations, and that these contributions are at least partially complementary.

## introduction how children acquire language from the environment is one of the fundamental mysteries of cognitive science. much theoretical, experimental, and computational research into this question has focused on acquiring abstractions over lowerorder symbols, such acquiring morphemes from phoneme sequences or syntactic structures from word sequences @xcite @xcite @xcite @xcite . children, however, do not get symbolic input; symbolic representations at any level of granularity constitute abstractions inferred from highly variable, noisy, and information-rich perceptual signals like audition and vision. this work joins a growing computational literature exploring the kinds of architectures and learning objectives that best support acquisition of linguistic representations directly from the speech signal without supervision @xcite . such models can be used to test questions about language acquisition under more realistic assumptions about the input signal, especially to the extent that they reflect known constraints on human cognition @xcite . this study uses computational modeling to examine two influential and possibly complementary ideas about how people learn abstract representations, including language, from data: learning to remember, and learning to predict. both hypotheses have been advocated by prior work in language acquisition, cognitive neuroscience, and computational modeling, yet their relative contributions to language learning are not yet clear. our model permits precise manipulation of memory and prediction pressures during acquisition, allowing direct comparison of these hypotheses. in so doing, we implement several constraints on real-time language processing that have not been simultaneously present in prior modeling of this domain: (1) we jointly segment and label the speech signal without supervision; (2) the learning objective is applied incrementally during real-time processing using only locally available feedback; results show a systematic improvement along multiple measures of phoneme induction quality from both learning to remember and learning to predict, suggesting that these two kinds of signals may play complementary roles during child language acquisition. the contributions of this work are as follows: ## model like many prior ann zero-resource speech processing models (e.g. @xcite @xcite , we employ an encoder-decoder framework. however, unlike previous approaches, our model decodes incrementally and hierarchically, with each layer decoding its inputs at their own timescale over a short window backward into the past and/or forward into the future. the model is thus required not only to describe the input signal (speech), but also its own sequence of latent representations (e.g. phones, words, etc.), much as people are implicitly thought to do in prior symbolic work on unsupervised language learning @xcite . our encoder model closely follows @xcite , and thus the primary technical contribution of this work lies in the cascaded incremental decoder and the layerwise incremental objective described below, both of which are designed to encourage repurposable segment representations based on locally available information. although encodings are ultimately the quantity of interest in unsupervised encoder-decoder models, prior work has shown that decoder design can be a major determinant of acquired representations @xcite . the overall design is schematized in figure 1 . code is available at https://github.com/coryshain/dnnseg . ## experimental design we assess the contribution of memory and prediction pressures to phoneme acquisition by (1) manipulating these pressures on models exposed to speech data from two unrelated languages (xitsonga and english) and ( 2 ) evaluating the effect of these manipulations on multiple measures of phoneme induction quality. ## results and discussion boundary and macro-averaged phoneme and feature classification f-measures from the bestperforming configuration on the english development set (@xmath0, @xmath1, @xmath2. english boundary performance (@xmath3.3) approaches previously reported unsupervised phoneme segmentation scores on different and therefore not directly comparable datasets @xcite @xmath4. the overall segmentation performance in xitsonga is considerably worse than that of english, consistent with prior evidence that word segmentation in the zerospeech 2015 xitsonga partition is harder than english (e.g. @xcite . by contrast, classification metrics in xitsonga are better than in english, which is again consistent with prior findings of stronger unsupervised classification performance in xitsonga @xcite . the difference in relative performance between segmentation and classification in the two languages could be due in part to differences in register: the english data is spontaneously produced, while the xitsonga data is read speech. longer average phoneme duration (100ms vs 70ms) and cleaner articulations in xitsonga could plausibly give rise to this asymmetry, and further investigation is left to future work. the model substantially outperforms the untrained baseline (u) on all metrics and outperforms the cross-language baseline (x) on all metrics but boundary f in xitsonga, which could be due in part to the larger size of the english-language training set. results therefore indicate that the reconstruction and prediction objectives have contributed to unsupervised discovery of phonemic patterns in both languages. segmentation and macro-averaged classification f-measures by language and experimental condition are given in figure 2 . results show a contribution of both memory (@xmath5, with a similar distribution of relative performance between the two languages, supporting the existence of language-general influences of prediction and memory on phoneme learning. as shown in figure 2 , models without memory pressures (@xmath6. there also appears to be a ceiling effect of backward reconstruction size, with a jump in performance at @xmath7. importantly, at layer 1, @xmath8, which falls within even conservative estimates of the storage duration of unanalyzed auditory traces in humans @xcite . the @xmath9. prediction pressures also support discovery of phoneme boundaries, as shown by the generally worse boundary performance of @xmath10. @xmath11. in addition, figure 2 shows that memory and prediction both modulate phoneme classification performance, with a roughly convex performance surface around a peak at @xmath12, @xmath13, @xmath14. a similar peak emerges in the feature classification results for english, along with a local feature classification peak in xitsonga for @xmath15. a 250ms auditory memory window thus supports both phoneme segmentation and classification in our models, with additional benefits from predicting over short intervals @xcite . for feature classification, the primary determinant of performance across languages is the prediction objective, with performance generally increasing up to @xmath16. there is also an effect of encoder depth in these results, such that encoders with more layers (@xmath17, despite the fact that all metrics reflect performance at the first layer. this result supports a contribution of multiscale modeling, even if the segmentation behavior at higher layers does not clearly correspond to a theory-driven level of representation (see section 4.3). figure 3 reports performance differences by metric against baseline u (untrained). training yields consistent and often substantial improvements across metrics in multiscale (@xmath18, but can fail to improve in the absence of these characteristics. memory and prediction therefore modulate not only absolute performance, but also the utility of language experience. figure 4 reports performance differences by metric against baseline x (cross-language). english segmentation is substantially helped by experience with (i.e. training on) english, especially under strong memory pressures. however, xitsonga segmentation is generally worse for the xitsongatrained model than the english-trained one. this might be due to the fact that the english training set is larger, and/or to low overall levels of segmentation performance in xitsonga. while we leave further investigation of this exception to future work, the classification metrics still show a clear benefit of in-domain training in both languages, but only in the presence of prediction pressures. the baseline x results bear on the degree to which speech processing patterns can plausibly be innately specified. although the set of phonological categories and features are classically regarded as universal @xcite , it is well known that the "same" phonological abstraction (e.g. voicing) can be phonetically cached out in different ways depending on the language (e.g. @xcite . our results suggest that, at least between xitsonga and english, this variation is both (1) constrained enough to permit recognition of non-trivial patterns from speech in other languages on the basis of general, possibly innate processing biases, and (2) substantial enough to give rise to a benefit of direct experience with the target language, even for language-general constructs like phoneme categories and phonological features. we use linear regression on the combined metrics to quantitatively evaluate the contribution of both memory and prediction pressures to phoneme acquisition. results show significant positive contributions to acquisition from memory pressures (@xmath19.006), prediction pressures (@xmath20.001), and multiscale encoding (@xmath21.001). 8 the boundary precision/recall trade-off illuminates the mechanisms by which memory and prediction pressures affect learning (figure 5 ). without memory pressures (@xmath22, segmentation rates are high, resulting in high recall and low precision. introducing memory pressures (@xmath23, resulting in a more balanced p/r trade-off. without prediction pressures (@xmath24, segmentation rates are generally low, resulting in higher precision and lower recall. introducing prediction pressures (@xmath25, again resulting in a more balanced trade-off. to understand this pattern, recall that a boundary in our model represents both a cost 8 see appendix g for details. (flushing the memory cell) and a benefit (injecting top-down feedback). the cost of forgetting is plausibly greater for reconstruction than prediction, since only the current layer has had direct access to the sequence of reconstruction targets. by contrast, the benefit of top-down feedback is plausibly greater for prediction than reconstruction, since the prediction can condition on contextual representations at multiple timescales. in our segmental model of speech processing, the objectives therefore induce countervailing biases that boost signal for phonological constructs, supporting their joint influence on phoneme acquisition from speech. ## conclusion we proposed an unsupervised deep neural model of speech processing that is incremental, segmental, and optimized by local feedback. we manipulated the model's objective function in order to investigate prior hypotheses about the role in human language acquisition of memory constraints on the one hand and predictive processing on the other. results support a role for both memory and prediction pressures for acquiring phonemes from speech. both objectives inform the model's segmentation behavior and the content of its segment encodings. in addition, results suggest that these two mechanisms coordinate to support phoneme discovery by introducing countervailing pressures toward retention of previously encountered signals (memory) and consultation of top-down signals (prediction). although our encoder model closely follows the definition in @xcite , it differs in the following ways: