Mapping Phonology to Semantics: A Computational Model of Cross-Lingual Spoken-Word Recognition (2022)

Abstract: Closely related languages are often mutually intelligible to various degrees. Therefore, speakers of closely related languages are usually capable of (partially) comprehending each other’s speech without explicitly learning the target, second language. The cross-linguistic intelligibility among closely related languages is mainly driven by linguistic factors such as lexical similarities. This paper presents a computational model of spoken-word recognition and investigates its ability to recognize word forms from different languages than its native, training language. Our model is based on a recurrent neural network that learns to map a word’s phonological sequence onto a semantic representation of the word. Furthermore, we present a case study on the related Slavic languages and demonstrate that the cross-lingual performance of our model not only predicts mutual intelligibility to a large extent but also reflects the genetic classification of the languages in our study.

## introduction speakers of closely related languages are usually capable of understanding each other's speech to a great degree without having a prior exposure to the second language (l2) or switching a lingua franca for communication 1 @xcite . the ability of the listener to comprehend spoken utterances in a different language (l2) using their native language (l1) competence is termed in the sociolinguistics literature as intercomprehension. @xcite categorized the factors that facilitate intercomprehension into linguistic factors (e.g., inherent crosslinguistic similarity between l1/l2) as well as extra-linguistic factors (e.g., listener's attitude towards communicating in a different language than their own l1). 1 a language used for communication between people who do not share a native language. several studies in the sociolinguistics literature have documented the levels of intercomprehension between related languages through empirical testing of mutual intelligibility with human subjects of different language backgrounds @xcite @xcite . it has been observed that objective measures of cross-language distance-such as lexical distance-are strong predictors of cross-linguistic intelligibility. therefore, mutual intelligibility of related languages is largely driven by the presence of word cognates-words that encode the same concepts with similar phonological forms across languages. from the psycholinguistic perspective, the listener's ability to recognize word forms in a different language is an example of the remarkable human ability to cope with the variability of speech @xcite . thus, spoken-word recognition across different, but related languages can be considered as lexical access problem-processing the spoken-word form to activate and retrieve the lexical category that is intended by the speaker. in the cognitive modeling literature, the task of spoken-word recognition has been addressed as a mapping problem between an acoustic-phonetic representation of the word form onto its semantic representation in memory (see @xcite for a detailed overview). recently, deep neural networks have been explored as models of spoken-word processing and recognition in several studies @xcite @xcite . our paper adds another contribution to this line of research by considering the cross-lingual aspects of spokenword recognition and sheds light on its contribution to cross-linguistic intelligibility using a computational model. our contribution is two-fold: (1) we present a neural model of spoken-word recognition and investigate the degree to which a monolingual model-i.e., has only been trained on a single language-is able to recognize the meaning of spo-ken words across related languages, and (2) we present a case study on the slavic languages which are remarkably similar and mutually intelligible to various degrees. concretely, we investigate the following research questions: rq1 does the cross-lingual performance of model predict the mutual intelligibility of the languages in our study? rq2 do the results of cross-lingual evaluation reflect the genetic relations among the studied slavic languages? rq3 which linguistic distance measures predict the cross-lingual performance of the monolingual models? and how do they compare to predictors of human performance? ## the model similar to the work of @xcite , our model takes a phonological sequence (spoken word form) as input, builds up a whole-word phonological representation of the sequence, and then projects it onto a semantic space (meaning representation) of the lexical item encoded by the word form. formally, we model the spoken-word recognition task of as a mapping function f θ : φ -→ s, where φ is the (discrete) space of phonological word forms, s is the word semantic space, and θ are the parameters of the mapping function. since phonological word forms can have any length, we model the function f using a recurrent neural network (lstm) followed by a multi-layer perceptron (mlp) (see figure 1 ). given the word form of the lexical category w as a phonological sequence here, d is the dimensionality of the semantic space. since our goal is to map the phonological input onto a semantic representation, the learning objective is based on vector regression loss and it aims to minimize the term ## experimental data in our paper, we present a case study on the slavic languages which have been shown to exhibit remarkable similarities and high degrees of mutually intelligibility at the conversational level @xcite @xcite . we use two languages of each of the three main branches of slavic languages, that is, russian and ukrainian for east slavic; polish and czech for west slavic; and bulgarian and croatian for south slavic foot_0 . one of the factors that drive our choice is the availability of high quality g2p tools available. ## evaluation during testing, the model computes the meaning representation of the phonemic sequence in the test language. to evaluate the model retrieval on the test set, the closest match between the model output and target vector for the model training language is retrieved using cosine similarity. cosine similarity determines whether two vectors are pointing in roughly the same direction and is measured by the cosine of the angle between two vectors. cosine similarity, on the abstract level, represents the proximity of the meaning retrieved by the listener to the actual meaning of the word. in other words, it would tell us how semantically similar two given vectors are. cosine similarity is computed between a model's output and all the 502 possible ground truth vector representations in the language of training. the vectors to be compared include all the word vectors used for monolingual testing. given these competing word embeddings, we also calculate average recall at 1 (r@1), recall at 5 (r@5), recall at 10 (r@10), as well as mean reciprocal rank (mrr) for the test data. r@n as the proportion of times that the set of top n word embeddings which are closest to the model's output also includes the ground truth vector representation. if the ground truth is most similar to the output vector of a model, r@1 is 1, otherwise it is 0. likewise, r@5 is 1, if the corresponding ground truth embedding is within the top 5 most similar words to the output vector, and r@10 is 1 if the embedding is within 10 most similar words. hence, the average r@n is a number between 0 and 1. the reciprocal rank information retrieval measure calculates the reciprocal of the rank at which the first relevant document was retrieved. for evaluation of the test data, we compute an average of reciprocal rank for all the given word forms. ## discussion and conclusion in this paper, we presented a spoken-word recognition model based on a recurrent neural architecture that maps variable-length phonological sequences of word forms into their respective meaning representations. our goal is to simulate auditory lexical processing in human listeners where we test the model on word forms from closely related languages and investigate the cross-lingual performance of the model. furthermore, we presented a case study on the family of slavic languages, which are known to be remarkable similar and exhibit (partial) mutual intelligibility to various degrees. we grounded our research on the findings from the sociolinguistics literature of slavic mutual intelligibility and intercomprehension. using our proposed model, we trained different instances of our model on six slavic languages: bulgarian, croatian, czech, polish, russian, and ukrainian. finally, we conducted a cross-lingual evaluation on our trained models to investigate their performance on retrieving and recognizing word forms from other l2 languages. returning to our research questions in §1, the cross-lingual analysis of our model performance has shown a trend where the model performance is better on languages that exhibit higher crosslinguistic intelligibility as documented in sociolinguistics studies (rq1). however, this effect is more consistent within south and east slavic languages, but less consistent in the case of west slavic languages (czech and polish). the factors that drive this inconsistency remain unknown and would require further future work to identify and analyze. despite this inconsistency, the clustering analysis on the cross-lingual concept retrieval performance resulted in a dendrogram that reflects the traditional genetic classification of the six studied slavic languages onto west, east, and south languages (rq2). furthermore, we have shown that cross-linguistic phonetic-lexical similarities