Familiar words but strange voices: Modelling the influence of speech variability on word recognition (2021)

Abstract: We present a deep neural model of spoken word recognition which is trained to retrieve the meaning of a word (in the form of a word embedding) given its spoken form, a task which resembles that faced by a human listener. Furthermore, we investigate the influence of variability in speech signals on the model’s performance. To this end, we conduct of set of controlled experiments using word-aligned read speech data in German. Our experiments show that (1) the model is more sensitive to dialectical variation than gender variation, and (2) recognition performance of word cognates from related languages reflect the degree of relatedness between languages in our study. Our work highlights the feasibility of modeling human speech perception using deep neural networks.

## introduction human speech is highly complex and variable. the sources underlying this variability include speakerrelated factors such as vocal tract shape, gender, age, and dialect as well as context-related factors such as word surprisal and phonological prominence. as a result, two acoustic realizations of the same word are unlikely to be identical even if produced by the same speaker. nevertheless, listeners can reliably recognize spoken words despite the lack of acoustic-phonetic invariance in speech @xcite . the robust human ability to decode the intended message from a highly variable, noisy speech signal enables speakers of different but related languages to communicate with each other using their own mother tongue -a phenomenon that has been referred to as receptive multilingualism @xcite . to gain a better understanding of human speech processing, a vast body of research at the intersection of speech perception and cognitive modeling has been dedicated to developing computational models of spoken word recognition (cf. @xcite for an overview). in a nutshell, models of spoken word recognition aim to simulate and explain the process of accessing the mental lexicon given a representation of an auditory word stimulus @xcite @xcite . despite the considerable differences in the representational specificity of the proposed models in the literature, there is a consensus among them with respect to the activation of multiple word candidates which leads to competition for lexical access @xcite . one model of word recognition that we take inspiration from in this paper is the distributed cohort model (dcm) @xcite , which is a connectionist model that defines the process of spoken word recognition as a mapping of low-level acoustic features onto the stored semantic and phonological representations, allowing efficient lexical access. a computational model of spoken word recognition allows researchers to simulate the conditions of behavioral experiments on human listeners and investigate whether the predictions of the model show human-like behavior. although deep neural networks (dnns) have become the dominant paradigm for automatic speech recognition (asr) research in the last decade @xcite @xcite , using dnn-based asr components to model human speech processing has only been explored recently with the earshot model @xcite . earshot is an incremental model based a long short-term memory (lstm) that captures the temporal structure of speech. the training data for the earshot model are spoken words produced using a speech synthesizer and each word is associated into a sparse vector that represents the word semantics. the authors use a unique but arbitrary sparse vector for each word, thus the semantic relatedness of words is not encoded in their representations. earshot is trained to map each acoustic word form onto its semantic vector. in this paper, we attempt to bridge between the connectionist view of word recognition and the recent advances in spoken language learning using deep neural networks. we also address some of the modeling limitations in the earshot model. precisely, our contribution is two-fold: (1) we propose a model of spoken word recognition based on a deep neural network that maps a spoken word form onto a distributed meaning representation. our model is trained on naturalistic data that consists of actual acoustic realizations of spoken words extracted from the german portion of the spoken wikipedia corpus. and (2) we investigate the degree to which the emergent representations from the model can generalize with respect to two sources of variability in speech signals -interspeaker variability and cross-lingual variability. ## a neural model of spoken word recognition our proposed model can be described at the high level as a function that maps the acoustic form of a word onto its lexical meaning. in the following, we describe the different representation schemes of our model. ## experiments we present and discuss the results of our experiments in this section. we first investigate the effect of different pre-trained word embeddings on the model performance. in the variability experiments, we aim to probe the model's ability to generalize to unheard speaker types as well as to recognize spoken word cognates in two languages that are phylogenetically related to german: dutch and english. following @xcite , we use the r @ n metric to evaluate our models for @xmath0, 5, 10}.