A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation (2024)

Abstract: Multilingual modelling can improve machine translation for low-resource languages, partly through shared subword representations. This paper studies the role of subword segmentation in cross-lingual transfer. We systematically compare the efficacy of several subword methods in promoting synergy and preventing interference across different linguistic typologies. Our findings show that subword regularisation boosts synergy in multilingual modelling, whereas BPE more effectively facilitates transfer during cross-lingual fine-tuning. Notably, our results suggest that differences in orthographic word boundary conventions (the morphological granularity of written words) may impede cross-lingual transfer more significantly than linguistic unrelatedness. Our study confirms that decisions around subword modelling can be key to optimising the benefits of multilingual modelling.

## introduction machine translation (mt) models have become increasingly multilingual @xcite . this greatly benefits low-resource languages through positive transfer from high-resource languages @xcite . however, increasing multilinguality in a limited shared parameter space can lead to suboptimal performance for high-resource languages @xcite . there is a tradeoff between maximising positive cross-lingual transfer (also known as synergy) while minimising negative cross-lingual interaction (also known as interference). several modelling decisions affect synergy and interference in multilingual mt. @xcite experimentally analysed the influence of factors like model size and language data proportions. one aspect their study failed to consider is subword segmentation. the shared subword vocabulary of multilingual models presents a similar trade-off as the shared parameter space -overlapping subword representations induce synergy, but having to represent multiple languages in a limited vocabulary can harm cross-lingual transfer @xcite @xcite . in this paper we experimentally analyse the role of subword segmentation in multilingual and crosslingual mt. our goal is to compare different classes of subword methods with regards to their ability to induce synergy, reduce interference, and transfer knowledge during cross-lingual finetuning. we also investigate how cross-lingual transfer is influenced by the linguistic similarities of interacting languages, with particular focus on factors related to subword structure like morphological typology and orthographic word boundary conventions (the degree to which morphemes are concatenated or written as separate orthographic words). we run experiments on translation from english to four linguistically diverse south african languages (see ## related work synergy and interference are well-established phenomena @xcite @xcite , but not well understood. @xcite address this by systematically analysing the role of several factors in synergy and interference: (1) model size, (2) data size, (3) language proportions, (4) number of languages, and (5) language relatedness. their results show that scaling model size and tuning the data sampling temperature greatly alleviates interference. they do not vary subword segmentation in their experiments, using the same sentencepiece @xcite vocabulary across all models. however, multilingual vocabularies are known to affect cross-lingual transfer through factors such as cross-lingual subword overlap @xcite @xcite and under-represented low-resource languages @xcite . these issues have mainly been studied for multilingual language modelling @xcite @xcite , but the same concerns hold for mt @xcite . we are unaware of existing work comparing the multitude of proposed subword methods in the context of multilingual mt. ## methodology this study involves two sets of mt experiments -(1) multilingual (trilingual) experiments to investigate synergy/interference, and (2) finetuning experiments to analyse cross-lingual transfer. our goal is to determine which subwords benefit low-resource languages and how cross-lingual transfer depends on linguistic typology. the linguistic diversity of south africa is an ideal testing ground for our purposes. siswati is a low-resource agglutinative language, so effective subword modelling is critical for dealing with the inevitably high proportion of out-of-vocabulary words in held-out datasets. we use siswati as the low-resource target language in our experiments and alternate the higher resourced language modelled alongside siswati between isi-xhosa, setswana, and afrikaans. while linguistic relatedness and morphological complexity are obvious features to consider in any analysis of cross-lingual interactions, we are unaware of work considering the impact of orthographic word boundary conventions. we include it as a factor in our study because of its potential relevance to subword segmentation. orthographic word boundaries determine the pre-tokenisation of text before subword segmenters are applied, so it could well affect aspects like segmentation granularity and overlap between the subword vocabularies of different languages. afrikaans is linguistically unrelated to siswati and also disjunctive, but because of its analytic morphology (lower morpheme-to-word ratio) its written words are sometimes more aligned to those of siswati (e.g. see phrases in this selection of languages allows us to isolate the cross-lingual effects of linguistic relatedness, morphological typology, and orthographic word boundary conventions. in the case of setswana-siswati, we can study whether the potentially positive cross-lingual effect of their linguistic relatedness is nullified by the fact that the two languages have very different conventions for orthographic word boundaries. ## experimental setup we compare five subword segmenters (four per experiment). we chose methods that represent the main paradigms of subword segmentation -deterministic segmentation, subword regularisation, learning subwords during training, and subword techniques for enhancing cross-lingual transfer. 1. bpe @xcite iteratively adds frequently co-occurring subwords to its vocabulary. we use it as a deterministic segmenter. 2. ulm @xcite learns segmentation to optimise a unigram language model and can be used as a probabilistic segmenter, exposing models to multiple subword segmentations for regularisation. we set the sampling parameter α to 0.5. 3. ssmt (meyer and buys, 2023) is a subword segmental mt model which learns subword segmentation jointly during mt training, with the goal of learning subwords that optimise mt performance. 4. obpe @xcite training we train models on wmt22 data @xcite and validate and test on flo-res @xcite . the number of training sentences are shown in for @xmath0, but for @xmath1, removing data size as an influence. we also removed examples from @xmath2. the hyperparameters of our models and subword methods are included in appendix a. ## results & discussion we plot the synergy/interference analysis of our multilingual experiments in figures 1 & 2 , while the absolute performance of the models are provided in ## conclusion we presented an in-depth study on the role of subwords in multilingual and cross-lingual mt. our results demonstrate that subword segmentation significantly influences cross-lingual interactions. ulm proves optimal for transfer to low-resource languages in multilingual modelling, while bpe enables greater cross-lingual transfer during finetuning. besides language relatedness, we show that similarities/differences in orthographic word granularity can greatly affect multilingual performance. there is more work to be done on the role of orthographic word boundary conventions in neural mt. future work could aim to design multilingual techniques that see past orthographic differences in order to leverage more fundamental similarities between languages.