Prompting with Phonemes: Enhancing LLM s’ Multilinguality for Non- L atin Script Languages (2025)

Abstract: Multilingual LLMs have achieved remarkable benchmark performance, but we find they continue to underperform on non-Latin script languages across contemporary LLM families. This discrepancy arises from the fact that LLMs are pretrained with orthographic scripts, which are dominated by Latin characters that obscure their shared phonology with non-Latin scripts. We propose leveraging phonemic transcriptions as complementary signals to induce script-invariant representations. Our study demonstrates that integrating phonemic signals improves performance across both non-Latin and Latin languages, with a particularly significant impact on closing the performance gap between the two. Through detailed experiments, we show that phonemic and orthographic scripts retrieve distinct examples for in-context learning (ICL). This motivates our proposed Mixed-ICL retrieval strategy, where further aggregation leads to our significant performance improvements for both Latin script languages (up to 12.6%) and non-Latin script languages (up to 15.1%) compared to randomized ICL retrieval.

## introduction large language models (llms) have demonstrated remarkable multilingual capabilities across various natural language processing (nlp) tasks. the increase in model parameters and rise of instruction datasets have led to the emergent capability of llms to perform tasks in few to zero shots @xcite @xcite or adapt efficiently to new tasks through in-context learning (icl) during inference @xcite . however, these capabilities remain disparate across languages @xcite , with one particular axis being along non-latin versus latin script languages @xcite @xcite . orthographic representation phonemic representation zho 阿桑奇是一名黑客活动家 esaŋt ͡ ɕʰiʂɻ̩ imiŋxeikʰexuotoŋt ͡ ɕia hin असां ज एक है कर-एि ट वस्ट हैं asaːɲd ͡ ʒe ekə ɦaeːkrə-ekʈivisʈə ɦaeːn arb ‫ھﺎﻛر‬ ‫ﻧﺎﺷط‬ ‫أﺳﺎﻧﺞ‬ aٔ saːnd ͡ ʒ naːʃtˤ haːkr jpn アサンジはハッカー活動家である 'ɑsɑnd ͡ ʒi wɑ haeɑka: kaetsədukə dɹ̩ u eng: assange is a hacker-activist to mitigate this disparity, we are motivated by the crucial role of phonemic awareness in human language acquisition and processing, facilitating skills like cross-lingual transfer and reading @xcite , in part due to cognates, borrowed words, and shared phonology between language families. we hypothesize that integrating phonemic information could also enable llms' robustness to the various choices of linguistic written script by capturing such alignments. for instance, in figure 1 , the japanese orthographic representation foot_0 for hacker ( ) is significantly different as compared to english. however, when observing the phonemic transcriptions-specifically, international phonetic alphabet (ipa) transcriptions at the level of phoneme discrimination-one could easily recognize the semantically similar words ( ) highlighted in green in figure 1 . on the other hand, the current prompting and icl schemes rely solely on orthographic text input, overlooking potentially valuable linguistic information encoded in the phonemic structure of language. while continual pretraining with phonemic data could enhance llms' multilingual capabilities, it faces several challenges. one significant limitation is the scarcity of large-scale multilingual datasets that align orthographic and phonemic tran-scriptions across diverse languages, especially for less-resourced languages. this lack of aligned data restricts the potential for fine-tuning models on phonemic information at scale. furthermore, pretraining with phonemic data demands substantial computational resources due to the increased data size and complexity, requiring extensive training time on high-performance infrastructure. hence, we propose the phonemic integration in llms via prompting and icl, as these promise a more flexible and resource-efficient approach to realize the integration's benefits. we hypothesize that augmenting with phonemic information could improve both demonstration retrieval and llm reasoning, by explicitly surfacing fundamental crosslinguistic information @xcite that textual scripts might not capture or may not be readily accessible in the llms' internal representations. our contributions include: phonemes are considered the smallest units of speech distinguishing one word (or word element) from another. 2 since llms are mostly trained on text, they might implicitly encode some information from the phonemes present in the text; however, investigative research on the phonemic awareness in language modeling is very limited. we discuss related work in phonemic awareness in nlp and other approaches to mitigating the performance divide of languages from different written scripts, motivating the need for our in-depth exploration of phonemic integration with llms. leveraging phonemes in text nlp. training text-based neural networks on both phonemic and orthographic information has given downstream task performance improvements in same-and crosslanguage nlp and speech tasks @xcite @xcite . however, such work was mostly limited to smaller lms or task-specific models (<1b parameters) where parameter constraints remain major potential obstacles for effective integration @xcite . on the other hand, as llms scale up to many billions of parameters, training-based schemes face limited orthographic-phonemic data and high computational costs; hence, motivating our work on phonemes integration with llms via prompting and enhanced icl. performance gaps in multilinguality. it was noted during the rise of early llms (<1b parameters) such as mbert @xcite and xlm-r @xcite ) that performance varied widely across languages @xcite , an observation that has persisted to modern lms (see references in section 1). some other script-related gaps include the greater difficulty of adapting to languages with non-latin scripts @xcite . non-phonemic approaches to bridging multilingual gaps via finetuning involved training schemes, translation, and contrastive presentations @xcite @xcite . with regards to script, while existing works explored improving transfer capability @xcite @xcite , they also remained limited to small lms (<2b parameters). bridging the gap in prompting and icl performance gaps involved cross-lingual chain of thought and demonstrations via translation @xcite , but have not to date involved phonology. ## is the written script sufficient for multilinguality? while the multilingual discrepancy performance when comparing non-latin and latin languages have been studied for smaller transformer-based lms (<1b parameters), these studies may not fully apply to the recent advancements in contemporary llms (≥ 7b parameters), thus requiring additional in-depth investigations. to empirically measure the gap in performance on latin versus non-latin scripts as a baseline for our work, we conduct an extensive and explicit (i.e., not post-hoc) study across 4 non-latin script languages -hindi (hin), arabic (arb), chinese (zho), japanese (jpn) foot_2 -and 6 latin script languages -german (deu), french (fra), dutch (nld), italian (ita), portuguese (por), spanish (spa). a comprehensive set of tasks, grouped by family, as presented in task metrics dataset name paws-x (yang et al., 2019) nlu accuracy xnli (conneau et al., 2018) aya-wiki (merity et al., 2022) nlg bleu @xcite aya-cnn @xcite mt chrf @xcite flores @xcite f1 aya-mlqa @xcite ) okapi-mmlu @xcite ) qa accuracy okapi-hellaswag @xcite okapi-arc @xcite we present results in figure 2 , finding that the performance of non-latin script languages remains inferior to latin script languages across families in this weight class. more importantly, while the performance gap is minor on nlu tasks such as paws-x and xnli across the majority of llms, the performance of non-latin script languages on nlg tasks such as aya-wiki is significantly worse compared to performance on latin script languages, especially english. for instance, the performance gaps between non-latin and latin languages are approximately 27.24, 23.08, 5.36 points on aya-wiki, aya-mlqa and flores with gemma-7b. with qwen2-7b, the gap on aya-wiki becomes even larger (approximately 28.59 points).we also observe the similar performance gap on aya-cnn task. however, as compared to aya-wiki, the performance across llms is significantly worse as the dataset contains more noise and long-context samples quickly going beyond the evaluated llms' context windows under fewshot settings. overall, these findings highlight that the non-latin vs latin language performance gap persists across llm families and to this day. to reduce this performance gap, we suggest incorporating phonemic information in model prompting, building on the insights from ziegler and goswami @xcite . their "psycholinguistic grain size theory" explains that learning to read depends on phonological awareness (the ability to recognize and work with sounds in language), and the complexity of a language's writing system determines how we process it-whether by focusing on letters, syllables, or whole words. based on this, we believe that adding phonemic information can help llms better process non-latin script languages, just as phonological awareness helps language learning for humans. ## prompting with phonemes motivated by insights from our pilot study (section 3), we now conduct experiments on represen-tative task categories where different llm families struggle to achieve similar performance between latin and non-latin languages, including nlg (aya-wiki), mt (flores), and question answering (aya-mlqa foot_3 ) and compare on the same metrics. due to the lack of publicly available multilingual corpora with orthographic-phonemic alignments, we adopt the approach of @xcite ; @xcite to construct our own aligned dataset for evaluation. specifically, we use epitran @xcite , a linguisticallycrafted tool, to generate ipa transcriptions for the orthographic-only multilingual datasets, setting up the foundation for our phonemic integration explorations with llms. in this work, we explore a series of experiments on phonemic integration with text-based llms and aim at improving llms' inference performance without the need for pretraining or fine-tuning. in our study, we focus on covering two contemporary mid-sized multilingual llms, including llama3-8b-instruct @xcite and qwen2-7b-instruct @xcite , chosen for their strong performance in our pilot study (figure 2 ). additional studies on the other two llm variants are provided in appendix b.1. more specifically, we explore two prompting approaches: direct prompting (section 4.1) and icl retrieval-based prompting (section 4.2). each approach is systematically applied to assess how well these strategies integrate phonemic information into the models and enhance llms performance. we then present results on latin and non-latin languages, showing ## analysis for consistency and conciseness, our analysis results are reported as averages over the aforementioned non-latin languages for the targeted tasks unless stated otherwise. how diverse are icl examples retrieved with script vs ipa? we present are both the script and ipa inputs needed for icl prompting? since ipa-icl and script-icl only consider the corresponding relevant information for selecting icl examples, (i.e., ipa and script respectively), the remaining information might be deemed unimportant and provide little information for llms' inference. for instance, if using script-icl, it is possible that only script information is essential while the ipa information can be left out. therefore, we conduct additional studies on the impact of removing this possibly unused information. as observed in does ipa-icl retrieval vary with different tokenizers? since text-based llm tokenizers might not have been trained with processing ipa inputs, we investigate the potential impact of different tokenizers on ipa-icl. as observed in ## conclusion we investigate the effect of integrating phonemic information towards improving the multilingual abilities of large-scale language models at inference time. our pilot study demonstrates that the performance gap between latin and non-latin script languages remains high even on latest state-of-the-art llms (with an approximate average 23% difference in performance) for certain tasks, especially on generation tasks (approx. 65%). motivated by these observations, we propose introducing phonemic integration in prompting with llms. we explore two incorporation mechanisms: direct prompting and icl retrieval. while we observe performance gains with direct prompting, our empirical study demonstrates that the icl retrieval provides an even more effective way to improve downstream task performance. the mixed-icl retrieval strategy captures diverse and similar icl examples beyond the textual scripts, leading to the best overall performance across multiple tasks, also evidenced in the case studies we present. we contribute an extensive empirical study on the effect of integrating phonemic information towards improving the performance of large-scale llms, reducing the performance gap between latin and non-latin performance. we show that incorporating phonemic information as ipa with few-shot icl retrieval prompting is an effective method to improve multilingual performance for languages with differing written scripts.