A phonetic model of non-native spoken word processing (2021)

Abstract: Non-native speakers show difficulties with spoken word processing. Many studies attribute these difficulties to imprecise phonological encoding of words in the lexical memory. We test an alternative hypothesis: that some of these difficulties can arise from the non-native speakers’ phonetic perception. We train a computational model of phonetic learning, which has no access to phonology, on either one or two languages. We first show that the model exhibits predictable behaviors on phone-level and word-level discrimination tasks. We then test the model on a spoken word processing task, showing that phonology may not be necessary to explain some of the word processing effects observed in non-native speakers. We run an additional analysis of the model’s lexical representation space, showing that the two training languages are not fully separated in that space, similarly to the languages of a bilingual human speaker.

## introduction compared to native speakers, non-native speakers perform differently in a variety of tasks related to auditory language processing, both at the phone and at the word level. at the phone level, these tasks usually require speakers to compare individual phones (e.g., phone discrimination or identification), while spoken word processing tasks usually test the implicit activation of a certain word in the memory (e.g., lexical priming, word translation). in some cases, non-native speakers' behavior is consistent across the tasks: lower performance in spoken word processing tasks is directly associated with difficult phone contrasts. for example, upon hearing a word rock, japanese speakers activate both rock and lock in their lexical memory @xcite , probably because they find it difficult to discriminate the english [ô]-[l] phone contrast @xcite . in other cases, however, non-native speakers' behavior in spoken word processing tasks cannot be explained by difficult phone contrasts @xcite @xcite . for example, in a translation task native english speakers may confuse russian words moloko [m@ë2"ko] ('milk') and molotok [m@ë2"tok] ('hammer'), even though this pair of words does not have a difficult phone contrast @xcite . this dissociation between the behavior in phone discrimination vs. spoken word processing tasks has been attributed to different kinds of representations involved. on the one hand, thanks to phonetic knowledge, speakers recognize individual phones in a given language. on the other hand, speakers store phonological representations of the words they know in their mental lexicon and use those representations to recognize spoken words (e.g., @xcite . it is often implicitly assumed that any lexical processing effect should be attributed to stored phonological representations of the words @xcite @xcite @xcite . at the same time, phonetic effects are not limited to the perception of individual phones: some existing theories argue that phonetic details are encoded in the lexical memory (e.g., @xcite @xcite . this raises the question: can some of the spoken word processing effects normally attributed to phonology be instead explained in terms of phonetic perception? in this study, we use computational modeling to test a hypothesis that some spoken word processing effects observed in non-native speakers can be explained without involving phonolexical representations-by phonetic perception, which results from phonetic learning, or speakers' attunement to the sounds of their native language @xcite . we use a model developed for speech technology applications @xcite . earlier, it was used to simulate early phonetic learning and successfully predicted some infant phone discrimination data @xcite . this model learns from natural speech data that is not segmented at the phone level. it never receives information about individual phones in isolation or about phone-level differences between words. therefore, the model is not equipped with an explicit mechanism to learn abstract phonolexical representations, making it a good candidate to test our hypothesis: if our model without the knowledge of phonology can correctly predict a particular effect, that effect can at least partially be attributed to phonetic learning. because our goal is to see whether phonetic perception can explain some of the existing data in principle, even just one phonetic model making correct predictions about the data would be a positive result. by design, lexical processing tasks require at least minimal knowledge of the target language. that is why they are normally carried out with bilingual speakers (or second language learners: @xcite @xcite . in bilingual speakers, the two languages interact at various levels, including lexical (e.g., @xcite . to take this into account, we simulate bilingual speakers by training the model on two languages simultaneously. we present three simulations. the first two show that the model exhibits predictable behaviors in discrimination tasks. in the third one, we present a case study to test whether a lexical processing effect commonly attributed to phonological representations can be explained in terms of phonetic learning alone, without the influence of phonol-ogy. in addition, we examine whether the representations in our bilingual model match the pattern observed in bilingual lexical access. existing studies @xcite @xcite show that upon the presentation of a word, competitor words in both languages may get activated (non-selective lexical access). we carry out a language classification task with our model, showing that the two languages are not fully separated in its representation space. ## simulation 1: phone discrimination previously, the monolingual cae-rnn was shown to correctly predict the crosslinguistic difference in the discrimination of the mandarin [c]-[tc h ] contrast @xcite , observed in mandarin-learning vs. english-learning infants @xcite . considering this, it may seem trivial to show that a model with some exposure to mandarin data (bilingual) would also achieve lower error than a model with no such exposure (english monolingual). however, potentially complex interactions between training languages may result in an unpredictable phonetic space. this is why we need to ensure the bilingual model behaves as expected, before we move on to the word-level tasks. to do this, we run a simple sanity check: whether a model trained on two languages behaves in a predictable way on the same mandarin contrast. ## simulation 2: word discrimination simulation 2 tests whether our bilingual model behaves in predictable ways at the word level. recall that it represents a sequence of any given duration as a fixed-dimensional vector. the compression of a dynamic speech sequence that unfolds in time into a 'static' vector results in information loss. since words are normally longer speech sequences than phones, it is not obvious whether the model's behavior in word discrimination and phone discrimination will be consistent. to examine this, we test the model on minimal pairs of words with [ô]-[l], a phone contrast on which the monolingual cae-rnn previously showed an infant-like crosslinguistic discrimination pattern @xcite . as before, we train the model on one or two languages and see if it behaves in a predictable way, this time in a word discrimination task. @xcite tested adult native speakers of american english and native japanese learners of english on the discrimination of english words rock-lock (i.e., [ô]-[l] contrast). learners with low english proficiency scored nearly at chance in this task, while highly proficient learners showed the discrimination scores close to those of native english speakers. this result is also in line with studies showing that native japanese speakers' discrimination of [ô]-[l] can improve after relevant phonetic training in english under certain conditions (e.g., @xcite @xcite . our goal is to test whether our model can correctly predict this word-level discrimination pattern. ## simulation 3: spoken word processing in this section, we present a case study to show how the model can be used to get a better understanding of spoken word processing. specifically, we are interested to know if some of the effects reported in the literature can be explained in terms of phonetic learning alone. difficulties with spoken word processing have been attributed to imprecise phonological encoding of non-native lexical representations @xcite @xcite @xcite , which results in a spurious activation of similarly sounding competitor words. to give an example from @xcite , if the word parent is encoded as [per@(n)t], with an optional [n], it may often be confused with parrot @xcite . we focus on one of the experiments in cook et al. in a translation judgment task, native russian speakers (proficient in english) and native english speakers (learning russian) heard a russian word (e.g., moloko [m@ë2"ko] 'milk') and then saw an english word (e.g., hammer, which translates into russian as 'molotok' [m@ë2"tok]). the participants had to decide if the english word was a good translation of the russian one. cook et al. manipulated the phone edit distance between the true translation and the competitor word: in the example above, the distance between [m@ë2"ko] and [m@ë2"tok] is 2. they found that non-native speakers made more mistakes than native speakers, and that increasing the phone edit distance between the target words decreased the size of this effect. they explain the effect by ambiguous ('fuzzy') non-native phonolexical representations. it is unclear, however, whether lexical phonology is necessary to explain the observed effect. to answer this question, we test whether our model with no access to phonology can correctly predict the described effect. clearly, our model does not know anything about word meanings and cannot be tested on a translation task. instead, we use a series of abx discrimination tasks to test whether the lower performance of non-native speakers can be explained in terms of acoustic embeddings of individual word tokens in the model's representation space. recall that the model has no access to phonology, and its acoustic embeddings result from phonetic learning alone. figure 4 shows the error rates of human participants in @xcite . the patterns that we focus on are: (1) lower error rate is associated with higher proficiency (exposure to russian) and higher 3 in @xcite . ## analyzing the model's representations most studies on bilingual lexical access advocate its non-selective nature: that is, speakers activate words in both languages in parallel, including in spoken word processing (e.g., @xcite @xcite . ideally, our model should show a similar pattern and not completely separate the two languages in its representation space. to examine this, russian-english ratio 75:25 50:50 25:75 10:90 mean 84.6 86.5 84.3 80.3 sd 1.0 0.9 0.4 1.6 using the bilingual models from simulation 3, we embed 5000 words per language. we then train a logistic regression classifier on 80% of this data to predict the language of a given word from its acoustic embedding, and test the classifier on the remaining 20% of words. the higher the accuracy of the classifier, the more linearly separable the two languages-specifically, their lexicons-in our model's representations. the results ( ## discussion we started by asking whether some of the difficulties in non-native spoken word processing can be explained at the level of phonetic perception, without involving phonolexical representations. to address this question, we presented a case study (simulation 3) with a computational model that learns from unsegmented speech data and does not have access to phonology. our model showed patterns similar to those found by @xcite in human speakers. this suggests that their results can be at least partly explained by phonetic learning. while we cannot estimate the relative contribution of the two factors-non-native phonetic perception vs. imprecise phonolexical representations-to the behavior of non-native speakers in the experiment of cook et al., we argue that both factors need to be considered as possible explanations of the spoken word processing difficulties in non-native speakers. note, however, that this result does not tell us whether the phonetic or the phonolexical explanation is more parsimonious-a question that should be addressed in the future. one could interpret our main result differently: that our model, in fact, has succeeded in learning phonological systems from speech data and cannot be considered a purely phonetic model. indeed, we know that deep neural networks can learn to encode various types of linguistic structure without explicit supervision (e.g., @xcite . in particular, speech models can achieve high accuracy in phone discrimination @xcite and classification @xcite , a finding sometimes interpreted as a successful acquisition of phonetic/phonological categories. while our model can discriminate at least some phone contrasts, too (simulation 1), this does not necessarily mean that it learns phonetic categories (see @xcite , for a relevant discussion). more importantly, what our model does not do is store explicit phonolexical representations in its memory, whereas the (imprecise) storage of word forms is one of the key premises of the phonolexical account explaining non-native speakers' difficulties in spoken word processing @xcite . therefore, we conclude that our results highlight the effects of phonetic perception on nonnative word processing. in simulation 1 and 2 we showed that our model trained simultaneously on two languages could correctly predict some phone-and word-level discrim-ination effects in infants and adults @xcite . this extends previous results on phone discrimination with monolingual model @xcite to word discrimination and to bilingual speakers. also, our analysis of model's representations indicates a substantial overlap between the lexicons of the two languages, mimicking non-selective lexical access in bilingual speakers (e.g., @xcite . all together, this suggests that the cae-rnn can be used as a tool to study not only native/non-native phonetic learning, but also native/non-native spoken word processing, including in bilingual speakers. our model helps to tease apart the potential impact of phonetic learning from other effects on spoken word processing. at the same time, it is not a cognitive model of the human mental lexicon, for example because it is devoid of semantics. a method to learn acoustic and semantic embeddings in parallel has been proposed in speech engineering @xcite , and future research could shed some light on whether this method can be used for studying human mental lexicon.