T ransli C o: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models (2024)

Abstract: The world’s more than 7000 languages are written in at least 293 scripts. Due to various reasons, many closely related languages use different scripts, which poses a difficulty for multilingual pretrained language models (mPLMs) in learning crosslingual knowledge through lexical overlap. As a consequence, mPLMs are faced with a script barrier: representations from different scripts are located in different subspaces, which can result in crosslingual transfer involving languages of different scripts performing suboptimally. To address this problem, we propose TransliCo, a framework that optimizes the Transliteration Contrastive Modeling (TCM) objective to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script (in our case Latin), which enhances uniformity in the representation space for different scripts. Using Glot500-m, an mPLM pretrained on over 500 languages, as our source model, we fine-tune it on a small portion (5%) of its training data, and refer to the resulting model as Furina. We show that Furina not only better aligns representations from distinct scripts but also outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks. Additionally, we achieve consistent improvement in a case study on the Indic group where the languages exhibit areal features but use different scripts. We make our code and models publicly available.

## introduction in recent years, mplms have made impressive progress in various crosslingual transfer tasks @xcite @xcite . such achievement is mainly due to the ## related work transliteration refers to converting languages from one script into another script @xcite . transliteration can increase lexical overlap, and therefore it has been shown to substantially improve the performance of neural machine translation for low-resource languages of different scripts @xcite @xcite . several studies also demonstrate that transliteration can enhance the crosslinguality of mplms across various dimensions. for instance, dhamecha et al. (2021) transliterate seven indo-aryan family languages into devanagari and show that common-script representations facilitate fine-tuning in a multilingual scenario. @xcite show that, by transliterating into latin, better performance is achieved when adapting mplms to new languages, particularly for low-resource languages. more recently, @xcite focus on indic languages and show that models directly pretrained on transliterated corpora in the latin script achieve better performance. however, the downside is that the model only supports one script and loses the ability to deal with the scripts in which the languages were originally written. this is not optimal when we expect predictions or generations in the original scripts. in contrast to this line of work, we aim to directly break the script barrier, instead of circumventing it by limiting the model to one common script. we use transliterations in our finetuning framework to improve the alignment across different scripts: the model after fine-tuning still supports the scripts it originally did. contrastive learning is a method for learning meaningful representations by contrasting positive pairs against negative pairs @xcite . this type of approach has achieved great success in learning visual representations @xcite @xcite . contrastive learning also demonstrates its effectiveness in nlp, especially for learning sentence representations @xcite @xcite . one major problem in contrastive learning is how to construct contrastive pairs. for a monolingual scenario, depending on specific downstream tasks, the positive pairs are usually constituted through data transformation or data augmentation strategies @xcite @xcite whereas the negative pairs are typically the remaining in-batch samples @xcite . in a multilingual scenario where parallel data is available, translations from different languages can be used to construct the positive pairs @xcite @xcite . unfortunately, large parallel corpora are mostly available for high-resource languages. therefore, aiming to improve crosslinguality, especially for under-represented languages, we start from the perspective of the script and construct positive pairs by using sentences in their original script and their latin transliterations that can be easily obtained. then we fine-tune an mplm with our contrastive framework translico. in this way, our work also resembles some post-pretraining alignment approaches @xcite @xcite that fine-tune a plm using token-level or sentence-level translations. ## methodology we present translico, a simple framework to address the script barrier by fine-tuning a plm on a small portion of the data that is used to pretrain the model. the framework consists of two training objectives: masked language modeling @xcite and transliteration contrastive modeling. we illustrate our framework in figure 2 and introduce our training objectives in the following. ## conclusion in this work, we propose a novel framework translico to fine-tune an mplm only using a small portion of sentences in its pretraining corpus and their latin transliteration. the framework contrasts the sentences in their original script and their transliterations to tackle the script barrier problem. using glot500-m as our source model, we fine-tune it using the proposed framework and present the resulting model: furina. through extensive experiments, we show furina better aligns the representations from different scripts into a common space and therefore outperforms glot500-m on a wide range of crosslingual transfer tasks. in addition, we conduct a case study on indic group languages that are known to be mutually influenced by each other but use different scripts. we show translico can also boost the performance for the indic group languages. we hope this framework can inspire more future work leveraging transliteration to improve crosslinguality of mplms.