Investigating Lexical Sharing in Multilingual Machine Translation for I ndian Languages (2023)

Abstract: Multilingual language models have shown impressive cross-lingual transfer ability across a diverse set of languages and tasks. To improve the cross-lingual ability of these models, some strategies include transliteration and finer-grained segmentation into characters as opposed to subwords. In this work, we investigate lexical sharing in multilingual machine translation (MT) from Hindi, Gujarati, Nepali into English. We explore the trade-offs that exist in translation performance between data sampling and vocabulary size, and we explore whether transliteration is useful in encouraging cross-script generalisation. We also verify how the different settings generalise to unseen languages (Marathi and Bengali). We find that transliteration does not give pronounced improvements and our analysis suggests that our multilingual MT models trained on original scripts are already robust to cross-script differences even for relatively low-resource languages.

## introduction as research in natural language processing (nlp) moves towards handling an increasing number of languages @xcite , one of the key challenges is targeting low-resource and morphologically rich languages @xcite . multilingual language models such as mbert @xcite and xlm-r @xcite ) have shown surprising cross-lingual ability in zero and few-shot scenarios for a diverse set of languages @xcite . in order for low-resource languages to optimally benefit from data available for related and higherresource languages, one research direction has been to explore what encourages better cross-lingual sharing in multilingual models, particularly in models that have joint vocabularies @xcite @xcite . one strategy for doing this is to preprocess the texts to reduce variation linked to differences in script and orthographic conventions, for example phonetisation, transliteration and transcription, in order to increase lexical overlap across languages. these pre-processing steps have been used in the literature across several multilingual nlp tasks @xcite @xcite @xcite @xcite . however, there is still some debate over how much transliteration helps in multilingual setups, despite it theoretically encouraging better lexical overlap, particularly for low-resource languages. for example, @xcite found that transfer may be helped by increased lexical overlap (although it also works without it) and @xcite argue that lexical overlap has a negligible impact on transfer. @xcite and @xcite found gains when transliterating, whereas for @xcite , results were less clear. in this study, we build on this previous work to further investigate how lexical overlap can help multilingual machine translation (mt) by taking as a case study several indian languages. figure 1 illustrates the degree of lexical overlap in the chosen languages of study: hindi, gujarati, nepali, bengali, and marathi. despite script differences, this example shows a sizeable amount of shared token overlap in terms of both characters and words. focusing on the translation of these languages (hindi, gujarati, nepali) into english, we explore the ideal parameter settings for multilingual mt (sampling vs. segmentation size) and look at how transliterating into a single script (i.e. gujarati into devanagari) may help performance. in addition, we look at how the trained models can transfer to other related languages (bengali and marathi) in zero-and few-shot settings. we find that transliteration does not significantly help performance in our multilingual mt setup, even for the lowestresourced language directions. our analysis suggests that even with relatively little data, the multilingual model trained on the original scripts seems to learn a sufficient mapping between original and transliterated tokens, possibly making transliteration redundant. even in zero-and few-shot transfer settings, we find only marginal improvements in the languages considered by using the multilingual model that uses transliteration as opposed to the multilingual model with the original scripts. ## related work multilingual models have been proposed for mt as well as other nlp tasks @xcite . within multilingual models, the promotion of lexical sharing has been the primary motivation to train multilingual models, which can especially aid low-resource languages @xcite . the choice of input unit has received a lot of attention, from the use of joint multilingual vocabularies @xcite @xcite and subword segmentation strategies @xcite to character-based @xcite and byte-based @xcite models. other works have explored phonetisation @xcite and transliteration/transcription in order to create a higher degree of lexical overlap in related languages that do not shared scripts @xcite @xcite @xcite . cross-lingual word embedding spaces have been of interest as well. @xcite map separately learnt embeddings to the same space, and other related works attempt to jointly learn a shared embedding space for multiple languages. crosslingual transfer studies on multilingual models such as mbert @xcite have also shown the utility of multilingual pre-training especially for zero-shot transfer @xcite . they show that overlap can lead to better zero-shot transfer, although there can still be transfer with no overlap, as also seen by @xcite . wu and dredze (2020) also see a positive correlation between lexical overlap and the zero-shot transfer performance. additionally, @xcite experiment with effect of shared vocabulary spaces in multilingual setups for several low-resource african languages (amharic, hausa, and swahili) and find that the number of languages used during pre-training has a positive effect on cross-lingual transfer only up to a certain point-which is improved by simply using a monolingual model with a multilingual tokeniser. variation in data availability, scripts, and morphosyntactic properties make adapting multilingual models to unseen languages challenging. transliteration, which directly encourages lexical overlap, has shown positive results for texts in different scripts @xcite . @xcite show that script plays a crucial role in improving transferability of multilingual models for languages that otherwise lag behind in performance. however, @xcite find that transcription (for slavic languages) degraded rather than aided performance, with the hypothesis that the high-resource setup made transcription unnecessary, especially given the noise introduced by transcription. in our work, we study the role of transliteration in the case of multilingual mt for a set of lower-resource language directions, using related indian languages with script differences. hindi, nepali, gujarati, bengali, and marathi are all indo-aryan languages, a sub-branch of the indo-european language family, with speakers primarily concentrated in the indian subcontinent. hindi (excluding urdu) foot_0 is spoken by approximately 340m l1 speakers (and 600m l1 or l2 speakers) and is considered to be the largest in terms of l1 speakers, whereas nepali, gujarati, bengali, and marathi have 16m, 57m, 272m, and 99m l1 speakers respectively. 3 hindi, nepali, and marathi share the same script (devanagari) and also certain morphosyntactic properties such as split ergativity and subject-object-verb word order with constraintbased reordering allowed. gujarati and bengali each use their own scripts, although they are still considered closely related to the other indo-aryan languages, with both lexical and grammatical similarities. in particular, in both languages there exist many words that are an exact phonetic match with hindi due to direct borrowing from sanskrit. due to these properties and the fact that the writing systems correspond well to the phonetic systems, transliteration from either the gujarati and bengali script into devanagari is mostly straightforward (see figure 3 for an example). ## experiments we study the effect of transliteration for multilingual mt to test the hypothesis that increased lexical overlap between the training languages could boost performance, particularly for lower-resourced language pairs. we study two different scenarios: (i) an in-language scenario, whereby we train and evaluate on the same set of language pairs, namely hindi (hi), nepali (ne), and gujarati (gu) into english, and (ii) zero-and few-shot transfer (via finetuning) of these models to two unseen related language pairs, namely marathi (mr) and bengali (bn) into english. we compare models trained on the original scripts and after transliteration (i.e. gujarati is transliterated into devanagari). since the aim of transliteration is to increase lexical overlap between the languages, we make sure to monitor for the degree of tokenisation, as well as data sampling, both crucial parameters in multilingual mt performance that directly affect token overlap, to ensure a fair comparison. ## results the main results are shown in ## conclusions in this work, we studied language sharing in multilingual mt of several languages in the indo-aryan language family (gujarati, nepali, and hindi into english). experimenting with sampling temperature and vocabulary size, we compare multilingual models using the original scripts and when transliterating gujarati into the same script as nepali and hindi (devanagari). surprisingly, even for the lowresource language directions (@xmath0, we find that transliteration is not particularly helpful. it seems that our multilingual models using the original scripts are able to correctly map phonetically equivalent tokens together, as suggested by (i) our analysis of the embeddings of identical characters across scripts and (ii) testing the robustness of the model to cross-script switching. finally, we test how well the models transfer to unseen related languages (marathi and bengali into english). we find that the model with transliteration does not perform significantly better with respect to generalisation to unseen languages, further supporting our previous findings.