R oman S etu: Efficiently unlocking multilingual capabilities of Large Language Models via R omanization (2024)

Abstract: This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages, specifically those using non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involve the continual pretraining of a English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches if not outperforms native script representation across various NLU, NLG and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP research.

## introduction large language models (llms) demonstrate remarkable proficiency across a broad spectrum of natural language processing (nlp) tasks, as evidenced by various studies @xcite @xcite @xcite . they excel not only † work done during internship at ai4bharat ‡ work done during employment at ai4bharat § corresponding authors: jaavid aktar husain (jaavidaktar@gmail.com), raj dabre (raj.dabre@nict.go.jp) and anoop @xcite in tasks for which they were explicitly trained but also in those for which they were not trained. this achievement is mainly due to the availability of corpora @xcite @xcite as well as the advancements in llms that leverage these datasets for pretraining @xcite @xcite . despite their proficiency in english, these models typically demonstrate reduced effectiveness when applied to non-english languages, highlighting a significant challenge in extending their benefits to non-english languages. the english-heavy llms @xcite @xcite still have some representation coverage from other languages due to data leakage while creating the pre-training dataset, particularly for languages that use the same script as english i.e., the roman script. this script sharing enables cross-lingual transfer and bestows some of these llm capabilities to these languages. for languages using non-latin scripts, the data representation is very limited to non-existent. tokenization of text in the languages exhibits high fertility @xcite and byte-level representation @xcite due to inadequate representation of these languages in the tokenizer vocabulary. hence, these llms perform poorly on most of these languages, and inefficient tokenization also leads to high inference latency as well as the inability to process long sequences. this disparity in performance raises a critical question: how can we extend the capabilities of llms to the languages written in non-latin scripts? a widely explored solution is the extension of the tokenizer vocabulary to incorporate new languages and continual pre-training on native language data @xcite @xcite . this approach is computationally demanding since the models need to for cpt, we show 3-shot results where available, or else 1-shot results. for ift, we use zero-shot results. in this work, we explore an alternative approach for more efficient knowledge transfer. instead of utilizing the native script, we use the romanized form of text as the interface to the llms in our approach romansetu. 1 the adoption of romanized representation is justified for several reasons. in many languages, romanized text has been frequently used in informal settings and on social media in recent history. this usage creates the potential for the inclusion of some romanized data representation in the pre-training corpus. additionally, code-mixing with english is a common occurrence and the romanized form shares tokens with english. this leads us to hypothesize that the romanized form is better aligned with english than the native script, thereby facilitating a more effective transfer from english. in our approach, we first continually pre-train an english llm like llama 2 @xcite on romanized text for a language generated via transliteration. subsequently, we perform instruction tuning on this continually pretrained model. we experiment with 5 languages from two language families and evaluate our approach on several nlu, nlg and mt benchmarks. our experi-ments and analysis indicate that: ## related work transliteration refers to the conversion of text written in one script to another. romanization is a specific instance of transliteration, where the target script is the roman/latin script. romanization has special significance since roman script is by far the most widely adopted script in the world and many language models are primarily trained for english, which is written in the roman script. transliteration is typically used to represent different languages in the same script to enable cross-lingual transfer. in nmt, @xcite show that transliteration shows improvements for lowresource languages with different scripts by transferring from related high-resource languages that use different scripts. @xcite show that transliteration helps even when only a contact relationship exists between the languages involved. @xcite show that transliteration during the pretraining stage for nmt also helps crosslingual transfer. transliteration has also been used for crosslingual transfer in the context of pretrained language models. some works @xcite @xcite employ transliteration to a common script during the pretraining phase to enable cross-lingual transfer. other studies @xcite @xcite adopt transliteration during the fine-tuning. transliteration could be done to a common non-latin script @xcite @xcite or to the latin script @xcite @xcite . while transliteration has been explored for language modeling as described above, our work differs from previous work in some important aspects: to the best of our knowledge, no prior research has investigated leveraging of romanization for mostly english llms in the context of crosslingual transfer to non-english languages. ## experimental settings we conducted comprehensive experiments spanning multiple languages, benchmarks and models. this section elaborates on the experimental settings. in all our experiments, we experiment with llama 2 7b @xcite as the base model under various settings. ## results in this section, we present the results of various experiments comparing romanized and native representations and discuss the results. we aim to address the following research questions: efficiency gains with romanization: to address this question, we compute fertility scores on the flores-200 dev set @xcite . fertility refers to the number of tokens per word generated by the llama 2 tokenizer @xcite . we compare the native script against two forms of romanized text: transliteration using indicxlit @xcite and rule-based extended itrans transliteration scheme implemented in the indicnlp library @xcite . ## conclusion in this study, we proposed the use of romanization to extend the performance of llms primarily trained in english to other languages. our approach successfully unlocks llm capabilities for non-english languages by using romanization to bridge english and non-english language representations. we have empirically demonstrated the effectiveness of this strategy through experiments involving few-shot prompting, continual pretraining, and instruction fine-tuning on a variety of tasks. additionally, using romanized data increases inference speed and maximum processable sequence length, while reducing memory requirements by 2x to 4x depending on the language.