Contact Relatedness can help improve multilingual NMT : M icrosoft STCI - MT @ WMT 20 (2020)

Abstract: We describe our submission for the English→Tamil and Tamil→English news translation shared task. In this submission, we focus on exploring if a low-resource language (Tamil) can benefit from a high-resource language (Hindi) with which it shares contact relatedness. We show utilizing contact relatedness via multilingual NMT can significantly improve translation quality for English-Tamil translation.

## introduction in recent years, neural machine translation @xcite @xcite @xcite (nmt) has become the most prominent approach to machine translation (mt) due to its simplicity, generality and effectiveness. in nmt, a single neural network often consisting of an encoder and a decoder is used to directly maximize the conditional probabilities of target sentences given the source sentences in an end-to-end paradigm. nmt models have been shown to surpass the performance of previously dominant statistical machine translation (smt) @xcite on many well-established translation tasks. however, in order to obtain good translation quality, nmt systems tend to require very large parallel training corpora @xcite . such corpora are not yet available for many language pairs. the indian subcontinent forms a linguistic area where languages from the dravidian and indo-aryan families have been in contact for a long time leading to significant sharing of vocabulary and a convergence of linguistic features @xcite . tamil is a major language from the dravidian language family spoken in southern india while hindi is a widely spoken indo-aryan language. @xcite estimate that lexical similarity between hindi and tamil to be around 27% in terms of character lcsr @xcite , while multiple works have shown that language representations of tamil and hindi cluster in the same neighbourhood in a multilingual vector space @xcite . while english-tamil parallel corpora is limited, more parallel corpora is available for english-hindi. in this paper, we explore if english-hindi can improve english-tamil machine translation due to the similarities between hindi and tamil on account of contact relatedness. to this end, we train multilingual nmt models for english-hindi and english tamil @xcite . previous work has explored whether high-resource languages can transfer knowledge to genetically-related low-resource languages @xcite . in contrast, we explore if contact relatedness can benefit low resource languages. we further explore if reducing the divergence between tamil and hindi data by representing them in the same script is beneficial. in addition, we explored target agreement models, tagged and noisy backtranslation in our submission. ## neural machine translation given a bilingual sentence pair (x, y), an nmt model learns its parameters θ by maximizing the log-likelihood p (y|x; θ), which is usually decomposed into the product of the conditional probability of each target word: p (y|x; θ) = m @xmath0, y 2 , .., y t-1 , x; θ), where m is the length of sentence y. an encoder-decoder framework @xcite @xcite is usually adopted to model the conditional probability p (y|x; θ). the encoder maps the input sentence x into a set of hidden rep-resentations h, and the decoder generates the target token y t at position t using the previously generated target tokens @xmath1. both the encoder and decoder can be implemented by different structure of neural models, such as rnn (lstm/gru) @xcite , cnn @xcite and selfattention @xcite . besides the basic component of the encoder and decoder, a sourcetarget attention mechanism @xcite is usually adopted to selectively focus on the source representations when generating a target token. the transformer @xcite model is the state-of-the-art nmt model relying completely on self-attention mechanism to compute representations of its input and output without using recurrent neural networks (rnn) or convolutional neural networks (cnn). in this work, we use the transformer architecture in all of our nmt models. we use smaller capacity networks compared to transformer-base, given the smaller size of the parallel data. ## multilingual learning for nmt the objective of multilingual learning for nmt is to construct a single model for translating to and from multiple languages. multilingual models can help improve performance of low-resource languages by transferring from high-resource related languages they are trained jointly with. @xcite introduced a many-to-many system, which still relied upon separate encoders and decoders for each language along with a shared attention mechanism. in contrast, @xcite introduced a "language flag"-based approach that shares the attention mechanism and a single encoder-decoder network to enable multilingual models. a language flag or token is part to the input sequence to indicate which direction to translate to. the decoder learns to generate the target given this input. this approach has been shown to be simple and effective and we use this in our multilingual models. as mentioned earlier, we train a joint hindi,tamil to english model as well as a joint english to hindi,tamil model. our hypothesis is that the contact relatedness between hindi and tamil will help transfer knowledge from hindi to tamil effectively. tamil and hindi use different scripts. however, it is possible to map almost all devanagari (the script used for hindi) characters to tamil characters. this mapping is deterministic but lossy since the tamil character set is smaller than the devanagari character set. such mapping will help to utilize the lexical similarity between the two languages directly. hence, we convert all hindi data to tamil script during a pre-processing step. we also report results of our multinmt models without script conversion of hindi to tamil. ## backtranslation backtranslation (bt) @xcite ) is a widely used data augmentation method where the reverse direction is used to translate sentences from target-side monolingual data into the source language. this synthetic parallel data is combined with the actual parallel data to re-train the model leading to better language modelling on the target-side, regularization and target domain adaptation. backtranslation is particularly useful for low-resource languages. we use backtranslation to augment our multilingual models. the backtranslation data is generated by multilingual models in the reverse direction, hence some implicit multilingual transfer is incorporated in the backtranslated data also. ## target agreement due to the autoregressive structure, current nmt systems usually suffer from the so-called exposure bias problem @xcite : during inference, true previous target tokens are unavailable and replaced by tokens generated by the model itself, thus mistakes made early can mislead subsequent translation, yielding unsatisfactory translations with good prefixes but bad suffixes. such an issue can become severe as sequence length increases. @xcite showed that the impact of this can be reduced by augmenting the training data with synthetic targets generated by a left-to-right (l2r) and a right-to-left (r2l) translation model. the directionality of the synthetic targets ensures that decoder input distribution becomes noisier (as happens at runtime) along one side of the target. the augmented data thus serves to reduce the divergence in the decoder input distribution. this is especially relevant to low-resource language scenarios where the model is not as robust to the decoder input distribution. ## results and discussion we report the sacrebleu scores on the dev sets and test sets provided in the wmt20 news translation task for both the language directions: @xmath2. we experimented with multilingual (multinmt), backtranslation (bt), noisy backtranslation (noisybt) and tagged backtranslation (taggedbt) and target agreement (ta) models. we observe that our multilingual models outperform the baseline bilingual models by significant margins. on the newstest2020 set, we see an improve of 2.9 and 1.5 bleu points respectively in the @xmath3. note that the gains translating into multiple targets is lower. however, when backtranslated data is added we observe an improvement of 2.3 bleu points in the @xmath4. note that the backtranslation data was generated via the multilingual @xmath5, hence there is an implicit benefit from multilinguality when using backtranslation. backtranslation also provides a good improvement in the @xmath6. we see that representing indian language data in the same script was very beneficial for @xmath7, while it did not help in the other direction. having disjoint vocabularies during generation possibly helps the model learn distinct language models for hindi and tamil. our results indicate that target agreement and noisy and tagged backtranslation schemes are not helpful in increasing the translation performance of the nmt models for the language pairs of our interest and requires more investigation on low resource language translation tasks. further analysis is needed to understand why backtranslation variants and target agreement did not show improvements in our setting. ## conclusion we believe contact relatedness can be utilized in the multilingual nmt framework for improving lowresource language translation. our initial results confirm this for english-tamil translation aided by english-hindi data. in addition, we show, that the popular data augmentation methods like backtranslation further helps in increasing the translation performance of multilingual nmt models.