Translating Similar Languages: Role of Mutual Intelligibility in Multilingual Transformers (2020)

Abstract: In this work we investigate different approaches to translate between similar languages despite low resource limitations. This work is done as the participation of the UBC NLP research group in the WMT 2019 Similar Languages Translation Shared Task. We participated in all language pairs and performed various experiments. We used a transformer architecture for all the models and used back-translation for one of the language pairs. We explore both bilingual and multi-lingual approaches. We describe the pre-processing, training, translation and results for each model. We also investigate the role of mutual intelligibility in model performance.

## introduction we present our findings from our participation in the wmt 2020 similar language translation shared task, which focused on translation between similar language pairs in low-resource settings. similar languages share a certain level of mutual intelligibility that may aid the improvement of translation quality. depending on the level of closeness, certain languages may share similar orthography, lexical, syntactic, and or semantic structures which may make translation more accurate. the level of mutual intelligibility is such that speakers of one language can understand another language without prior instruction in that other language. they can also communicate without the use of a lingua franca which is a link or vehicular language used for communicating between speakers of different languages @xcite . it is important to mention that, sometimes, the level of intelligibility varies in both directions. for instance, slovene -croatian intelligibility is said to be asymmetric such that speakers or slovene can understand spoken and written croatian better than speakers of croatian understand slovene @xcite . machine translation of similar languages has been explored in a number of works @xcite @xcite . this can be seen as part of a growing need to develop models that translate well in low resource scenarios. the goal of the current shared task is to encourage researchers to explore methods for translating between similar languages. we also view the shared task as useful context for studying interaction between degrees of similarity and mutual intelligibility on the one hand, and model performance on the other hand. we explore the use of bilingual and multilingual models for all the 5 shared task language pairs. we also perform back-translation for one language pair. in the remainder of this paper, we discuss related literature in section 2. we explain the methodology which includes a description of the transformer model, back-translation and beam search in section 3. in section 4, we describe the models we developed for this task and we discuss the various experiments we perform. we also describe the architectures of the models we developed. then we discuss the evaluation procedure in section 6. evaluation is done on both the validation and test sets. we conclude with discussion and the insights gained from this task in section 7. ## related work translation between similar languages has recently attracted attention. different approaches have been adopted using state-of-the-art techniques, methods, and tools to take advantage of the similarity between languages even in low resource scenarios. approaches that have been effective for other ma-chine translation tasks have proven to achieve success in the context of similar language translation as well. nmt models, specifically the transformer architecture, has been shown to perform well when translating between similar languages @xcite . the use of in-domain data for fine-tuning has also proven to be of remarkable benefit for this task. this problem has also been tackled both by using character replacement to leverage the orthographic and phonological relationship between closely related mutually intelligible language pairs @xcite . a new approach was also introduced for this task using a two-dimensional method that assumes that each word of the target sentence can be explained by all the words in the source sentence @xcite . within the realm of mt for low resource languages, recent work has focused on translation using large monolingual corpora due to the scarcity of parallel data for many language pairs @xcite @xcite . these approaches have leveraged careful initialization of the unsupervised neural mt model using an inferred bilingual dictionary, sequence-to-sequence language models, and back-translation to achieve remarkable results. the bilingual dictionary is built without parallel data by using an unsupervised approach to align the monolingual word embedding spaces from each language @xcite . since parallel data is not available in sufficiently large quantities, backtranslation is used to create pseudo parallel data. the monolingual data of the target language is translated into the source using an existing translation system (e.g., one trained with available gold data). the output is then used to train a new mt model @xcite . weak supervision caused by back-translation results in a noisy training dataset. this eventually can affect translation quality. more recent works adopt different approaches to manage noise in back-translation. for instance, phrase based statistical mt models are introduced as a posterior regularization during the back-translation process to reduce the noise and errors of the data generated @xcite ). another method @xcite uses cross lingual word embeddings incorporated with sub-word information. the weights of the log-linear model is then tuned through an unsupervised process and the entire system is jointly refined in opposite directions to improve performance. this method outperforms previous sota model with about 5-7 blue points. a re-scoring mechanism that re-uses the pre-trained language model to select translations generated through beam search has also been found to improve fluency and consistency of translations @xcite . yet another approach, combines cross-lingual embeddings with a language model to make a phrase-table @xcite . the resulting system is then used to generate a pseudo parallel corpus with which a bilingual lexicon is derived. this approach can work with any word or cross-lingual embeddings techniques. ## methodology motivated by the success of transformers and backtranslation, we develope a sequence-to-sequence approach using the transformer architecture perform back-translation for one language pair. for decoding, we use beam search (bs). bs is an heuristic decoding strategy based on exploring the solution space and selecting a sequence of words that maximize the overall likelihood of the target sentence. during the translation, we hold a beam of β sequences (beam size) which are iteratively extended. at each step, β words are selected to extend each of the sequences in the beam, so the output is β 2 candidate sequences (hypotheses), we retain only the β highest score hypotheses for the next step (top-β candidates) @xcite . in all our experiments we use beam size of 5 whilst decoding. ## data we used all the parallel data for all language pairs http://www.statmt.org/wmt20/similar. html . the task was constrained so we did not add any additional data to develop our models. we used the monolingual data for the sl-hr language pair for back-translation. ## evaluation we evaluated both the dev and test sets. we used the best-checkpoint metric with bleu score to evaluate the validation set at each iteration. we used a beam size of 5 during the evaluation . we de-tokenized from bpes back into words. ## conclusion we described our contribution to the wmt2020 similar languages translation shared task. we developed both bilingual and multilingual models for all pairs, in both directions. we showed back-translation to help improve performance on one pair. we also showed how mutual intelligibility between a pair of languages ( measured by jaccard similarity) positively correlate with model 2 we multiplied the jaccard similarity by 100 to reduce the range of values on the y axis. performance (in bleu). future work can focus on exploiting other similarity metrics and providing a more in-depth study of mutual intelligibility between similar languages and how it interacts with mt model performance both in bilingual and multilingual models. the utility of back-translation on pairs we have not studied can also be fruitful.