How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings (2021)

Abstract: How do neural networks “perceive” speech sounds from unknown languages? Does the typological similarity between the model’s training language (L1) and an unknown language (L2) have an impact on the model representations of L2 speech signals? To answer these questions, we present a novel experimental design based on representational similarity analysis (RSA) to analyze acoustic word embeddings (AWEs)—vector representations of variable-duration spoken-word segments. First, we train monolingual AWE models on seven Indo-European languages with various degrees of typological similarity. We then employ RSA to quantify the cross-lingual similarity by simulating native and non-native spoken-word processing using AWEs. Our experiments show that typological similarity indeed affects the representational similarity of the models in our study. We further discuss the implications of our work on modeling speech processing and language similarity with neural networks.

## introduction mastering a foreign language is a process that requires (human) language learners to invest time and effort. if the foreign language (l2) is very distant from our native language (l1), not much of our prior knowledge of language processing would be relevant in the learning process. on the other hand, learning a language that is similar to our native language is much easier since our prior knowledge becomes more useful in establishing the correspondences between l1 and l2 @xcite . in some cases where l1 and l2 are closely related and typologically similar, it is possible for an l1 speaker to comprehend l2 linguistic expressions to a great degree without prior exposure to l2. the term receptive multilingualism @xcite has been coined in the sociolinguistics literature to describe this ability of a listener to comprehend utter-ances of an unknown but related language without being able to produce it. human speech perception has been an active area of research in the past five decades which has produced a wealth of documented behavioral studies and experimental findings. recently, there has been a growing scientific interest in the cognitive modeling community to leverage the recent advances in speech representation learning to formalize and test theories of speech perception using computational simulations on the one hand, and to investigate whether neural networks exhibit similar behavior to humans on the other hand @xcite @xcite @xcite @xcite . in the domain of modeling non-native speech perception, @xcite have shown that a neural speech recognition system (asr) predicts japanese speakers' difficulty with the english phonemic contrast /l/-/ô/ as well as english speakers' difficulty with the japanese vowel length distinction. @xcite have shown that a model of non-native spoken word processing based on neural networks predicts lexical processing difficulty of english-speaking learners of russian. the latter model is based on word-level representations that are induced from naturalistic speech data known in the speech technology community as acoustic word embeddings (awes). awe models map variable-duration spoken-word segments onto fixed-size representations in a vector space such that instances of the same word type are (ideally) projected onto the same point in space @xcite . in contrast to word embeddings in natural language processing (nlp), an awe encodes information about the acoustic-phonetic and phonological structure of the word, not its semantic content. although the model of @xcite has shown similar effects to what has been ob-native encoder language ( ) non-native encoder language ( ) non-native encoder language ( ) a simulation of a native speaker's phonetic perceptual space a simulation of a non-native speaker's phonetic perceptual space a simulation of a non-native speaker's phonetic perceptual space spoken-word stimuli from native speakers of language ( ) figure 1: an illustrated example of our experimental design. a set of n spoken-word stimuli from language λ are embedded using the encoder f (λ) which was trained on language λ to obtain a (native) view of the data: x (λ/λ) ∈ r d×n . simultaneously, the same stimuli are embedded using encoders trained on other languages, namely f (α) and f (β) , to obtain two different (non-native) views of the data: x (λ/α) and x (λ/β) . we quantify the cross-lingual similarity between two languages by measuring the association between their embedding spaces using the representational similarity analysis (rsa) framework. in this paper, we present a novel experimental design to probe the receptive multilingual knowledge of monolingual awe models (i.e., trained without l2 exposure) using the representational similarity analysis (rsa) framework. in a controlled experimental setup, we use awe models to simulate spoken-word processing of native speakers of seven languages with various degrees of typological similarity. we then employ rsa to characterize how language similarity affects the emergent representations of the models when tested on the same spoken-word stimuli (see figure 1 for an illustrated overview of our approach). our experiments demonstrate that neural awe models of different languages exhibit a higher degree of representational similarity if their training languages are typologically similar. ## proposed methodology a neural awe model can be formally described as an encoder function f : a -→ r d , where a is the (continuous) space of acoustic sequences and d is the dimensionality of the embedding. given an acoustic word signal represented as a temporal sequence of t acoustic events @xmath0, a 2 , ..., a t ), where a @xmath1, an embedding is computed as here, θ are the parameters of the encoder, which are learned by training the awe model in a monolingual supervised setting. that is, the training spoken-word segments (i.e., speech intervals corresponding to spoken words) are sampled from utterances of native speakers of a single language where the word identity of each segment is known. the model is trained with an objective that maps different spoken segments of the same word type onto similar embeddings. to encourage the model to abstract away from speaker variability, the training samples are obtained from multiple speakers, while the resulting awes are evaluated on a held-out set of speakers. our research objective in this paper is to study the discrepancy of the representational geometry of two awe encoders that are trained on different languages when tested on the same set of (monolingual) spoken stimuli. to this end, we train awe encoders on different languages where the training data and conditions are comparable across languages with respect to size, domain, and speaker variability. we therefore have access to several encoders {f (α) , f (β) , . . . , , f (ω) }, where the superscripts {α, β, . . . , ω} denote the language of the training samples. now consider a set of n held-out spoken-word stimuli produced by native speakers of language λ: a first, each acoustic word stimulus in this set is mapped onto an embedding using the encoder f (λ) , which yields a matrix x (λ/λ) ∈ r d×n . since the encoder f (λ) was trained on language λ, we refer to it as the native encoder and consider the matrix x (λ/λ) as a simulation of a native speaker's phonetic perceptual space. to simulate the phonetic perceptual space of a non-native speaker, say a speaker of language α, the stimuli a (λ) 1:n are embedded using the encoder f (α) , which yields a matrix x (λ/α) ∈ r d×n . here, we read the superscript notation (λ/α) as word stimuli of language λ encoded by a model trained on language α. thus, the two matrices x (λ/λ) and x (λ/α) represent two different views of the same stimuli. our main hypothesis is that the cross-lingual representational similarity between emergent embedding spaces should reflect the acoustic-phonetic and phonological similarities between the languages λ and α. that is, the more distant languages λ and α are, the more dissimilar their corresponding representation spaces are. to quantify this cross-lingual representational similarity, we use centered kernel alignment (cka) @xcite . cka is a representationlevel similarity measure that emphasizes the distributivity of information and therefore it obviates the need to establish the correspondence mapping between single neurons in the embeddings of two different models. moreover, cka has been shown to be invariant to orthogonal transformation and isotropic scaling which makes it suitable for our analysis when comparing different languages and learning objectives. using cka, we quantify the similarity between languages λ and α as here, sim(λ, α) ∈ [0, 1] is a scalar that measures the correlation between the responses of the two encoders, i.e., native f (λ) and non-native f (α) , when tested with spoken-word stimuli a (λ) 1:n . sim(λ, α) = 1 is interpreted as perfect association between the representational geometry of models trained on languages λ and α while sim(λ, α) = 0 indicates that no association can be established. likewise, we obtain another nonnative view x (λ/β) of the stimuli a (λ) 1:n using another non-native encoder f (β) . we then quan-tify the cross-lingual representational similarity between languages λ and β as if sim(λ, α) > sim(λ, β), then we interpret this as an indication that the native phonetic perceptual space of language λ is more similar to the non-native phonetic perceptual space of language α, compared to that of language β. note that while cka(., .) is a symmetric metric (i.e., cka(x, y) = cka(y, x)), our established similarity metric sim(., .) is not symmetric (i.e., sim(λ, α) : = sim(α, λ)). to estimate sim(α, λ), we use word stimuli of language α and collect the matrices x (α/α) and x (α/λ) . then we compute when we apply the proposed experimental pipeline across m different languages, the effect of language similarity can be characterized by constructing a cross-lingual representational similarity matrix (xrsm) which is an asymmetric m × m matrix where each cell represents the correlation (or agreement) between two embedding spaces. ## acoustic word embedding models we investigate three different approaches of training awe models that have been previously introduced in the literature. in this section, we formally describe each one of them. ## representational similarity analysis figure 3 shows the cross-lingual representational similarity matrices (xrsms) across the three different models using the linear cka similarity metric. 3 warmer colors indicate a higher similarity between two representational spaces. one can observe striking differences between the pge-cae models on the one hand, and the cse model on the other hand. the pge-cae models yield representations that are cross-lingually more similar to each other compared to those obtained from the contrastive cse model. for example, the highest similarity score from the pge model is sim(rus, bul) = 0.748, which means that the representations of the russian word stimuli from the bulgarian model f (bul) exhibit the highest representational similarity to the representations of the native russian model f (rus) . on the other hand, the lowest similarity score from the pge model is sim(pol, deu) = 0.622, which shows that the german model's view of the polish word stimuli is the view that differs the most from the view of the native polish model. likewise, the highest similarity score we observe from the cae model is sim(bul, rus) = 0.763, while the lowest similarity is sim(deu, bul) = 0.649. if we compare these scores to those of the contrastive cse model, we observe significant cross-language differences as the highest similarity scores are sim(pol, rus) = sim(por, rus) = 0.269, while the lowest is sim(bul, deu) = 0.170. this discrepancy between the contrastive model and the ## conclusion we presented an experimental design based on representational similarity analysis (rsa) whereby we analyzed the impact of language similarity on representational similarity of acoustic word embeddings (awes). our experiments have shown that awe models trained using decoding objectives exhibit a higher degree of representational similarity if their training languages are typologically similar languages. we discussed our findings from a typological perspective and highlighted pre-lexical features that could have an impact on the models' representational geometry. our findings provide evidence that awe models can predict the facilitatory effect of language similarity on cross-language speech perception and complement ongoing efforts in the community to assess their utility in cognitive modeling. our work can be further extended by considering speech segments below the word-level (e.g., syllables, phonemes), incorporating semantic representations into the learning procedure, and investigating other neural architectures.