Predicting non-native speech perception using the Perceptual Assimilation Model and state-of-the-art acoustic models (2021)

Abstract: Our native language influences the way we perceive speech sounds, affecting our ability to discriminate non-native sounds. We compare two ideas about the influence of the native language on speech perception: the Perceptual Assimilation Model, which appeals to a mental classification of sounds into native phoneme categories, versus the idea that rich, fine-grained phonetic representations tuned to the statistics of the native language, are sufficient. We operationalise this idea using representations from two state-of-the-art speech models, a Dirichlet process Gaussian mixture model and the more recent wav2vec 2.0 model. We present a new, open dataset of French- and English-speaking participants’ speech perception behaviour for 61 vowel sounds from six languages. We show that phoneme assimilation is a better predictor than fine-grained phonetic modelling, both for the discrimination behaviour as a whole, and for predicting differences in discriminability associated with differences in native language background. We also show that wav2vec 2.0, while not good at capturing the effects of native language on speech perception, is complementary to information about native phoneme assimilation, and provides a good model of low-level phonetic representations, supporting the idea that both categorical and fine-grained perception are used during speech perception.

## introduction our native language has an impact on our speech perception. importantly, it influences the way we perceive unfamiliar sounds-for example, from other languages. for instance, japanese native speakers have difficulties in telling apart the english sounds @xcite , which are not in their native phoneme inventory, and confuse the english words 'right' and 'light' @xcite , while this contrast is highly distinct for english native speakers. the influence of native language on speech perception is not yet fully understood. a simple approach might assert that two unfamiliar sounds are only discriminable if they would be classified as distinct phonemes by the listener. however, some unfamiliar sound contrasts are easy. for example, english-speaking listeners have no difficulty with zulu click contrasts @xcite or the thai [w]-[7] vowel contrast @xcite . the perceptual assimilation model (pam: @xcite proposes that non-native speech perception is governed by the complete profile of listeners' assimilations to their native phoneme categories, which may be indeterminate. thus, even if english has neither [w] nor @xcite , english-speaking listeners do not assimilate @xcite to any specific english vowel, while they categorise @xcite as english @xcite . since one of the two unfamiliar phones is not categorised as any native phoneme but the other is, listeners are able to discriminate them from one another. pam strongly implies that speech sound discrimination relies on phoneme-level representations cleansed of rich phonetic detail. however, an alternate approach proposes that listeners rely only on fine-grained, rich phonetic representations which are fine-tuned to the statistics of the native language, rather than mental categories @xcite @xcite . recent modelling results by @xcite have been interpreted as support for the view that, at least in early infancy, phoneme categories are not essential to explain effects of native language on speech perception. however, while there is evidence supporting each of these ideas, previous work has not directly compared quantitative predictions of pam against the predictions of a fine-grained phonetic account. we make use of the overlap score, an operationalization of pam proposed by @xcite which yields quantitative predictions of the degree of discriminability of an arbitrary pair of non-native phones from the results of an assimilation task performed on these phones. we then rely on recent developments in statistical machine learning of speech audio representations without the use of phoneme labels @xcite . we generate fine-grained statisticallydriven representations of the experimental stimuli using a dirichlet process gaussian mixture model @xcite , the model proposed by @xcite as an account of early phonetic learning. we also test wav2vec 2.0 @xcite , a more recent speech representation model. we compare these as predictors on a novel, open speech perception dataset containing french and english participants' behavioural results for 61 different vowels in brazilian portuguese, german, turkish, estonian, french and english. our goal is to assess which model predicts participants' discriminability behaviour the best for the two groups, and, which one is able to predict the difference of behaviour between the two groups, presenting a native language effect. our results show that phoneme categories cannot be replaced by even the state of the art in fine-grained statistical phonetic modelling. while both the pam-based predictor and the wav2vec 2.0 model are good predictors of the discriminability of stimuli, there is a small but consistent advantage for the assimilation-based predictor. furthermore, the fine-grained models do not at all account for the differences in discriminability between the french and english listener groups, but, instead, seem to be improved predictors of a language-neutral "phonetic similarity." the two factors, phonemic assimilation and phonetic similarity, are complementary, and both have a role to play in speech perception, consistent with other evidence showing that listeners are sensitive to fine-grained acoustic details at both neural and behavioural levels @xcite . the code and data we use are openly available. foot_0 ## discrimination dataset to compare the predictions of pam and finegrained acoustic models, we create an experimental dataset that measures french-and englishspeaking listeners' ability to discriminate native and non-native vowels. participants judged stimuli constructed from twelve brazilian portuguese vowels ( ). these vowels were chosen to have a large coverage of the vowel space using a small number of languages, as well as to maximise the differences in perception behaviour between the french-speaking and the english-speaking listeners, as many front rounded and nasal vowels are included. to test listeners' phone discrimination ability, we use an abx discrimination task. the participant hears a stimulus a, a stimulus b, then a third one, x. they then have to decide which stimuli, a or b, is the closest to x, which always belongs to the same phone category as either a or b. participants' accuracy reflects the distinctness of the a and b stimuli for the participants. in our case, french and english-speaking listeners' discrimination of pairs of native and non-native vowel sounds were tested, with a, b and x always belonging to the same language and using the same consonant frame (more details about the stimuli in section 5). the results of the two groups were used to compare the relative predictive power of the pam-type predictor versus the fine-grained model representations. ## pam: from assimilation to discrimination according to the perceptual assimilation model @xcite participants' behaviour during an assimilation task of non-native sounds can predict the discriminability of pairs of these same non-native sounds by listeners with the same language background. to derive such a predictor this, we conduct a separate experiment which elicits how the vowels listed in the previous section are assimilated by french-and english-speaking participants to their native vowel categories (see section 3.1). we apply the logic of pam using the overlap score proposed by @xcite : see section 3.2. ## fine-grained phonetics: predicting discrimination directly from audio we compare the experimentally derived negoverlap score against several types of fine-grained phonetic representations from models (see sections 4.2 and 4.3) as predictors of listeners' discrimination behaviour. for each of the experimental stimuli, we calculate these representations for a, b, and x, and compute a score called ∆ (see section 4.4), which measures how discriminable the representations are. they are considered more fine-grained than the neg-overlap score since they are more detailed in time (multiple representational frames per second compared to one assimilation judgement for a whole stimulus) and in dimensionality (around a thousand compared to around ten for the vote vectors used by the neg-overlap score). ## methods: discrimination and assimilation experiments in this section we provide basic information about the experiments with human participants. for more details, see the appendix. ## results we compare the neg-overlap score with ∆ scores from fine-grained phonetic models. we seek to assess how well they can predict the results of the discrimination experiment. this comparison is done using two metrics: the log-likelihood of a binary regression model applied to the results of the experiment, and the spearman's ρ correlation between the average of the given predictor and listeners' accuracy for each pair of phones. we calculate the log-likelihood (ℓℓ) using a probit model, fitted to the complete set of discrimination responses for each listener group, with correct responses coded as 1 and incorrect responses as 0. we fit one model for each of the neg-overlap and ∆ scores we seek to compare, with the given score as a predictor: each observation is paired with the score for the stimulus heard by the participant. in the case of neg-overlap, the score is calculated for the pair of phones using the results of the assimilation experiment, while the ∆ scores are calculated on the a, b, and x audio files. in addition to a global intercept, the model has additional predictors accounting for effects of a number of nuisance factors: whether the right answer was a (1) or b (0); the position of the trial in the experimental list; and a categorical predictor for the participant. higher (less negative) log-likelihood values indicate better predictions. we complement the log-likelihood with a correlation statistic, which has the advantage of being on the interval between 0 and 100%, and having been used in previous works @xcite . consistent with @xcite , we calculate the correlation at the level of phone pairs, rather than individual items, foot_2 and we use spearman's ρ, a correlation between ranks: in this case, the rank of the discrimination accuracy, versus the rank of the neg-overlap or average ∆ score, for the given phone pair. to assess the statistical significance of the differences between model comparison scores, we perform a bootstrap resampling of the participants' results: for each item used in the discrimination experiment, we resample the results with replacement in order to obtain five scores. we generate n bootstrap samples. for each sample, we calculate ℓℓ (ρ) for all predictor scores, and then compute the difference in ℓℓ (ρ) to compare two predictors. if the 95 % bootstrap confidence interval of the differences does not contain zero, we take the difference to be significant. ## discussion and summary we have presented a dataset of non-native speech perception combining a native phoneme assimilation task with a discrimination task, appropriate for exploring questions about the role of native language categories in speech perception. the data is open, and contains a large number of vowels, presented to french-and english-speaking listeners. our analysis shows that french and english speakers' patterns of assimilation to native vowel phonemes are good predictors of discriminability by naive listeners, consistent with the perceptual assimilation model. the prediction, however, is not perfect. we then did a comparable analysis using rich, fine-grained phonetic models obtained by applying machine learning to speech, which do not directly use coarse-grained phoneme categories. we showed that none of these models is consistently better at predicting listeners' discrimination behaviour than the experimentally-derived assimilation scores, and, contrary to previous studies, they show only minimal effects of the "native" language on which they were trained. the assimilation scores, in contrast, are good at capturing the differences between our french-and english-speaking listeners. finally, we demonstrate that the native wav2vec model is a good model of universal phonetics, and provides information about phonetic similarity which is complementary to the phoneme assimilation-based scores, consistent with the very old idea that discrimination experiments are explained by a mix of categorical effects and phonetics @xcite . compared to previous attempts to measure phonetic similarity, the wav2vec model does not require choosing specific phonetic measures in advance (like formants), and is applicable to any speech stimuli. language carrier sentence triphone's transcription volunteers american english i like [...] here orth. for reference words, ipa for the rest 4f, 4m brazilian portuguese eu digo [...] aqui ipa precised, orth. in sentences 2f, 3m estonian ma ütlen [...] siin orth. 3f, 2m german (munich) ich sage [...] hier ipa 3f, 3m metropolitan france's french je dis [...] ici orth., except for oe, ø, o and o 4f, 4m