Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer (2024)

Abstract: Approaches to improving multilingual language understanding often struggle with significant performance gaps between high-resource and low-resource languages. While there are efforts to align the languages in a single latent space to mitigate such gaps, how different input-level representations influence such gaps has not been investigated, particularly with phonemic inputs. We hypothesize that the performance gaps are affected by representation discrepancies between those languages, and revisit the use of phonemic representations as a means to mitigate these discrepancies.To demonstrate the effectiveness of phonemic representations, we present experiments on three representative cross-lingual tasks on 12 languages in total. The results show that phonemic representations exhibit higher similarities between languages compared to orthographic representations, and it consistently outperforms grapheme-based baseline model on languages that are relatively low-resourced.We present quantitative evidence from three cross-lingual tasks that demonstrate the effectiveness of phonemic representations, and it is further justified by a theoretical analysis of the cross-lingual performance gap.

## introduction large language models have significantly advanced natural language processing, offering improved capabilities across numerous languages. however, substantial performance gaps remain, particularly between high-resource languages like english and the majority of the world's low-resource languages. while these gaps are partly driven by discrepancies in data availability and quality, recent studies suggest that linguistic gaps-potentially caused by structural and lexical differences-also contribute significantly to these disparities . cross-lingual transfer techniques, which aim to adapt to arbitrary target language, have shown promise with the advancement of pre-trained multilingual language models @xcite @xcite . however, they continue to face challenges, particularly with lowresource languages. one line of prior research has focused on mitigating these gaps through crosslingual representation alignment @xcite @xcite , but these efforts often overlook the impact of varying input representations on performance consistency across languages. in this work, we explore the use of phonemic representations written in international phonetic alphabet (ipa) characters as a robust input representation (see figure 1 ) to reduce linguistic gaps and, consequently, performance gaps across languages. we define the linguistic gap as the representation discrepancy between embedding vectors and the performance gap as the relative difference in downstream task performances between languages, to analyze the impact of phonemic representations in cross-lingual adaptation. our empirical analysis shows that phonemic representations consistently reduce linguistic gaps between languages compared to orthographic character-based models. this reduction in linguistic gaps directly correlates with smaller performance gaps in tasks such as cross-lingual natural language inference (xnli), named-entity recognition (ner), and part-of-speech (pos) tagging, demonstrating the potential of phoneme-based models to enhance cross-lingual transfer across diverse languages. we further support these findings with theoretical analysis from domain generalization literature, where we frame the performance gap as a consequence of linguistic gaps driven by lexical and syntactic differences. our key contributions are as follows: ## experimental setup in this section, we describe the experiment setup in terms of models, datasets, and downstream tasks, including the selected target languages and details for preprocessing. additionally, we provide details on evaluation strategies, particularly on quantifying the performance and linguistic gap. ## results and analysis here, we present our observations and analyses of the results. we first discuss the behavior of phoneme-based model towards low-resource languages and writing systems, which contributes to robust cross-lingual performance. next, we delve into the performance and linguistic gaps of phoneme-based models through empirical and theoretical analyses. ## conclusion towards robust multilingual language modeling, we argue that mitigating the linguistic gap between different languages is crucial. moreover, we advocate the use of ipa phonetic symbols as a universal language representation partially bridges such linguistic gaps without any complicated cross-lingual training phase. empirical validation on three representative nlp tasks demonstrates the superiority of phonemic representation compared to subword and character-based language representation in terms of the cross-lingual performance gap and linguistic gap. theoretical analysis of the cross-lingual performance gap explains such promising results of phonemic representation.