Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non- L atin Scripts (2025)

Abstract: Decoder-only large language models (LLMs) excel in high-resource languages across various tasks through few-shot or even zero-shot in-context learning (ICL). However, their performance often does not transfer well to low-resource languages, especially those written in non-Latin scripts. Inspired by recent work that leverages transliteration in encoder-only models, we investigate whether transliteration is also effective in improving LLMs’ performance for low-resource languages written in non-Latin scripts. To this end, we propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script transliteration, or (3) combined. We apply these methods to several representative LLMs of different sizes on various tasks including text classification and sequential labeling. Our findings show that the effectiveness of transliteration varies by task type and model size. For instance, all models benefit from transliterations for sequential labeling (with increases of up to 25%%).

## introduction decoder-only llms, such as llama @xcite , @xcite , xglm @xcite , and bloom @xcite , have shown impressive capability across a wide range of tasks for high-resource languages, particularly through few-shot icl @xcite . however, they often underperform in low-resource languages, especially those written in underrepresented scripts. multiple reasons exist, such as the scarcity of low-resource languages in the training data @xcite , insufficient crosslingual alignment during pretraining @xcite , as well as english being the only language in the instruction tuning phase 1 we consider a special type of transliteration that converts non-latin scripts into latin script (also called romanization). ( @xcite . the mainstream methodology attempts to address this issue by translating the texts written in languages other than english into english using either external machine translation systems @xcite or self-translate, i.e., translation by leveraging the few-shot translation capabilities of the model itself @xcite . however, the quality of translations is constrained by the quality of the external systems or the llm itself. additionally, this type of approach is infeasible for truly low-resource languages. recent studies have demonstrated that leveraging transliteration into a common-script effectively improves the crosslingual transfer performance of encoder-only models on low-resource languages of non-latin scripts @xcite . this is because a common script facilitates the model to transfer knowledge through increased lexical overlap @xcite @xcite . inspired by this line of work, a natural research question is to explore whether transliteration is also effective for decoderonly llms, especially through their icl capability the topic of the news वे ि@xmath0कन क' रात क' प,हल/ हाथ बटाई धोखे बाज़ 8नकलस बै क;<ॉम ?वारा गे म जीतय वाले गोल का साथ र ◌ी। with its latin transliteration: oveckin kii raat kii pahilii haath battaaii dhokhebaaja nikalas baikasttrom dvaaraa gem jiitaya vaale gol kaa saath rahii. is label: {} to this end, the paper investigates the above research question and proposes three types of prompt templates where the non-latin targetlanguage text is represented in (1) its original script (script {orig} ), (2) latin script (script {latn} ), or (3) both (script {combined} ). given that texts in different scripts convey the same semantics, the knowledge encoded in one script should complement the other. a capable model, therefore, should leverage this complementarity: when a word or an entire sentence in the original script is not well understood, the model should refer to its transliteration, and vice versa. we apply our methods to several llms on various tasks and observe that the effectiveness of transliteration varies by task type and model size. transliteration is particularly helpful for sequential labeling. on other tasks, however, transliteration-augmented prompts are less effective, indicating models might have limited capacity to exploit complementary information. our contributions are as follows: (i) we conduct the first investigation towards the effectiveness of transliteration in icl for decoder-only llms. (ii) we propose transliteration-augmented prompts that are specifically for low-resource languages in non-latin scripts; (iii) we offer insights on when and how transliteration can enhance icl performance. ## experimental settings models. we experiment with six models: llama2-7b @xcite , mistral-7b-instruct @xcite , and the 7b, 3b, 1b, and 560m variants of the bloom model @xcite . llama2 is a model trained on 28 languages and 5 scripts (cyrillic, latin, hang, hani and japanese). mistral is an englishcentric model trained on five languages in latin script, while bloom is a multilingual llm covering a wide range of languages in 11 scripts. 2 we select these models to compare the effectiveness of transliteration-augmented icl on model type (english-centric vs multilingual models) and model size (different variants of bloom). ## results and discussion we report the average performance across all languages in ## related work positive effects of transliterating data into a common script have been demonstrated in various recent works for encoder-only models @xcite @xcite . additionally, leveraging transliteration as an auxiliary input at fine-tuning stage improves the cross-script performance @xcite . to improve icl performance for low-resource languages, demonstrations play an important role. one line of approaches replaces the target-language texts with english translations @xcite @xcite . another type of research augments the icl demonstrations, e.g., by retrieving the most similar english texts to the target-language text @xcite @xcite 5 conclusion this study explores the effectiveness of transliteration in enhancing the icl performance of decoderonly llms, focusing on low-resource languages written in non-latin scripts. by proposing three prompt templates -using original script, latin script, and a combination of both -we evaluate their impact across various tasks on several representative llms. our findings indicate that transliteration is particularly effective for sequential labeling but its benefits for text classification tasks are less consistent. we also observe a mixed effect of transliteration related to the model type and model size. our results highlight the potential of transliteration as a possible way to enhance llms' performance for low-resource languages.