Alternative Input Signals Ease Transfer in Multilingual Machine Translation (2022)

Abstract: Recent work in multilingual machine translation (MMT) has focused on the potential of positive transfer between languages, particularly cases where higher-resourced languages can benefit lower-resourced ones. While training an MMT model, the supervision signals learned from one language pair can be transferred to the other via the tokens shared by multiple source languages. However, the transfer is inhibited when the token overlap among source languages is small, which manifests naturally when languages use different writing systems. In this paper, we tackle inhibited transfer by augmenting the training data with alternative signals that unify different writing systems, such as phonetic, romanized, and transliterated input. We test these signals on Indic and Turkic languages, two language families where the writing systems differ but languages still share common features. Our results indicate that a straightforward multi-source self-ensemble – training a model on a mixture of various signals and ensembling the outputs of the same model fed with different signals during inference, outperforms strong ensemble baselines by 1.3 BLEU points on both language families. Further, we find that incorporating alternative inputs via self-ensemble can be particularly effective when training set is small, leading to +5 BLEU when only 5% of the total training data is accessible. Finally, our analysis demonstrates that including alternative signals yields more consistency and translates named entities more accurately, which is crucial for increased factuality of automated systems.

## introduction machine translation has seen great progress, with improvements in quality and successful commercial applications. however, the majority of this improvement benefits languages with large quantities of high-quality training data (high-resource languages). recently, researchers have focused on the development of multilingual translation models @xcite capable of translating between many different language pairs rather than specialized models for each translation direction. in particular, such multilingual models hold great promise for improving translation quality for low-resource languages, as grouping languages together allows them to benefit from linguistic similarities as well as shared data between related languages. for example, training a translation system with combined assamese and bengali data would enable transfer learning between the two languages. we investigate how to enable multilingual translation models to optimally learn these similarities between languages and leverage this similarity to improve translation quality. the fundamental unit representing lingual similarity is the token -languages that are similar often have similar words or phrases -and during training, translation models can learn strong representations of tokens in low-resource languages if they are also present in high-resource languages. however, a challenge arises when similar languages share only a small amount of tokens, which inhibits the transfer to limited and trivial cases of token sharing, e.g., punctuation marks and digits. this is particularly clear in cases where similar languages are written in different scripts, as the amount of shared tokens is small compared to languages using the same written script. an example would be hindi and gujarati, which have phonetic similarity but are written in their own native scripts. to tackle inhibited transfer due to distinct writing systems, we transform the original input via transliteration, the process of converting text from one script to another, to get alternative signal from the original source sentences. transliteration has the romanized version and '__bn_inscrip__' the same input but written in the script of hindi, a language within the same language family). the log probabilities output by the model given each type of input are averaged for subsequent decoding process. ## method multilingual translation models enable languages to learn from each other, meaning low-resource languages can benefit from similarities to highresource languages where data is plentiful. however, surface-level differences between languages, such as writing system, can obscure semantic similarities. we describe an approach to transliterating input sentences to various alternative forms that maximize transfer learning between different languages, and various modeling approaches to incorporating such varied inputs. ## experimental setup dataset we train our model on two language families: indic and turkic. the indic dataset is from the wat multiindic mt task foot_1 , including 10 indic languages and in total around 11 million indic-english bi-texts. six of the indic languages are indo-aryan languages and the rest are dravidian languages. all of these languages use a different writing system. the turkic dataset is collected from the open parallel corpus @xcite foot_2 . for relatively high-resourced language turkish, we randomly select 4 million subset from the ccaligned (el-kishky et al., 2020) corpus. within this dataset, two languages use cyrillic alphabet (kazakh and kyrgyz) and the rest use latin alphabet. detailed dataset statistics are displayed in single-input model to test the effectiveness of each input signal, we train models on each single type of input: original input (base), phonetic input (ipa), romanized input (romani) or input all expressed in the script of a language within the same language family (inscrip). on the indic dataset, multi-source self-ensemble similar to above, we train a single model on the mixture of original input and one of {ipa, romani, inscrip} input for multi-source self-ensemble. to enforce fair comparisons with the ensembled models, which have more parameters in total, we train two sizes of the self-ensemble (se) model, one having the same size of a single baseline model, the other scaled to have twice the number of parameters of a single base model. ## results in this section, we compare the performance of our proposed multi-source self-ensemble model to various alternative ways of input combinations on two low-resource language families: indic and turkic languages. furthermore, we show multisource self-ensemble learns faster and generates more consistent and accurate translations. ## conclusion to overcome the low token-overlap issue exhibited in multilingual mt systems due to distinct writing system, we examined three alternative signals (phonetic, romanized and in-family transliterated input) and investigated four approaches (input concatenation, multi-encoder, multi-source ensemble, self-ensemble) to combining them with the original input. our results show that training a single model with a mixture of diverse signals and performing self-ensemble during inference time can improve bleu by 1.3 points on indic and turkic dataset. the improvements can reach +5 bleu when training data size is small. further, we show this approach generate more accurate and consistent translation of named entities which greatly impacts the factuality accuracy of news translation.