Does Transliteration Help Multilingual Language Modeling? (2023)

Abstract: Script diversity presents a challenge to Multilingual Language Models (MLLM) by reducing lexical overlap among closely related languages. Therefore, transliterating closely related languages that use different writing scripts to a common script may improve the downstream task performance of MLLMs. We empirically measure the effect of transliteration on MLLMs in this context. We specifically focus on the Indic languages, which have the highest script diversity in the world, and we evaluate our models on the IndicGLUE benchmark. We perform the Mann-Whitney U test to rigorously verify whether the effect of transliteration is significant or not. We find that transliteration benefits the low-resource languages without negatively affecting the comparatively high-resource languages. We also measure the cross-lingual representation similarity of the models using centered kernel alignment on parallel sentences from the FLORES-101 dataset. We find that for parallel sentences across different languages, the transliteration-based model learns sentence representations that are more similar.

## introduction in the last few years, we have seen impressive advances in many nlp tasks. these advances have been primarily led by the availability of large representative corpora and improvement in the architecture of large language models. while improving model architectures, training methods, regularization techniques, etc., can help advance the state of nlp in general, the unavailability of large, diverse corpora is the bottleneck for most languages @xcite . thus to inclusively advance the state of nlp across languages, it is crucial to develop techniques for training mllms that can extract the most out of existing multilingual corpora. here, we focus on the issue of diverse writing scripts used by closely related languages that may prevent mllms from learning good cross-lingual representations. previous papers @xcite have noted that low-resource languages that use unique scripts tend to have very few tokens representing them at the tokenizer. as a result, these languages tend to have more unknown tokens, and the words in these languages tend to be more split up by subword tokenizers. often we can easily transliterate from one script to another using rule-based systems. for example, there are established standards that can be used to transliterate greek (iso 843), cyrillic (iso 9), indic scripts (iso 15919), and thai (iso 11940) to the latin script. in this paper, we focus on the indic languages, which have the highest script diversity in the world. many south asian and southeast asian languages are intimately connected linguistically, historically, phonologically @xcite and phylogenetically. however, due to different scripts, it is difficult for mllms to fully exploit this shared information. among the indic languages we considered in this study we encounter eleven different scripts. these are shown in we empirically measure the effect of transliteration on the downstream performance of mllms. we pretrain albert (base, 11m parameters) @xcite and @xcite @xcite models from scratch on indic languages. we pretrain two variants of each model, one with the original writing scripts and the other after transliterating to a common writing script. henceforth, we will refer to the transliterated script model as uni-script model and the other as a multi-script model. we evaluate the models on downstream tasks from the indicglue benchmark dataset @xcite . in order to rigorously compare the two models, we finetune using nine random seeds on all downstream tasks. then we perform the mann-whitney u test (mwu) between the uni-script and multi-script models. using the mwu test, we conclude that transliteration significantly benefits the low-resource languages without negatively affecting the comparatively high-resource languages. we also measure the cross-lingual representation similarity (clrs) to understand why the uniscript model performs better than the multi-script model. to measure the clrs, we use the centered kernel alignment (cka) @xcite similarity score. we measure the cka similarity score between the hidden representations of the models on the parallel sentences of the indic languages from the flores-101 dataset @xcite . we find that, compared to the multi-script models, the uni-script models achieve a higher cka score, and it is more stable throughout the hidden layers of the models. based on this, we conclude that the uni-script models learn better cross-lingual representation than the multi-script models. in summary, our contributions are primarily three-fold: our code is available at github 1 and our model 1 https://github.com/ibraheem-moosa/ xlm-indic weights can be downloaded from hf hub foot_0 foot_1 foot_2 foot_3 . ## cross-lingual representation similarity in this section, we analyze why the uni-script model performs better than the multi-script model from the perspective of cross-lingual representation similarity. following @xcite , @xcite and @xcite we apply cka to measure clrs. we use the cka implementation from the ecco library @xcite . we use parallel sentences on thirteen languages from the flores-101 @xcite dataset. for the albert models, which are trained on only the indo-aryan languages, we only consider panjabi, hindi, bengali, oriya, assamese, gujarati, marathi, and nepali sentences. for the rembert models, we additionally consider kannada, telugu, malayalam, tamil, and english sentences. first, we calculate the sentence embeddings of these parallel sentences from the models. sentence embedding is calculated by averaging the hidden state representations of the tokens. then, we calculate the cka similarity score between the sentence embeddings for each language pair. for each language, we average its cka similarity scores. in figure 1 we plot this average cka similarity for each layer of the models. we see that clrs score drops significantly at the last layer for all models. however, the uni-script models retain high clrs score until the eleventh layer, whereas the multi-script models have low clrs score from the ninth layer. overall the clrs score of the uni-script models are more stable. this indicates that the uni-script models have learned better cross-lingual representations. ## tokenizer quality analysis in terms of performance, we expect the transliteration model to exploit better tokenization across the languages. following @xcite and @xcite , we measure the subword fertility (average number of tokens per word) and the ratio of words unbroken by the tokenizer. from figure 2, we can see that transliteration reduces the splitting of words. this indicates that many words that were represented by different tokens in the multi-script model are represented by a single token in the transliteration model. on average, the albert uni-script tokenizer has a lower subword fertility score of 1.55 compared to the multi-script tokenizer's 1.825. the uni-scirpt tokenizer also has a lower proportion of continued word score of 0.36 while the multi-script tokenizer has a score of 0.45. ## conclusion and future work in this paper, we show that transliterating closely related languages to a common script improves multilingual language model performance and leads to better cross-lingual representations. we conducted rigorous statistical analysis to quantify the significance and effect size of transliteration on downstream task performance. we found that transliteration especially improves performance on comparatively low-resource languages and did not hurt the performance on high-resource languages. this findings are in agreement with @xcite . our results indicate that in other scenarios where closely related languages use different scripts, transliteration can be used to improve the performance of language models. for example, slavic and turkic languages present similar scenarios. we would like to extend our study to models at different scales and more languages in the future. also, another interesting future direction would be to just use the transliteration for pretraining signal but give the model the ability to deal with the original scripts.