Multilingual Neural Machine Translation (2020)

Abstract: The advent of neural machine translation (NMT) has opened up exciting research in building multilingual translation systems i.e. translation models that can handle more than one language pair. Many advances have been made which have enabled (1) improving translation for low-resource languages via transfer learning from high resource languages; and (2) building compact translation models spanning multiple languages. In this tutorial, we will cover the latest advances in NMT approaches that leverage multilingualism, especially to enhance low-resource translation. In particular, we will focus on the following topics: modeling parameter sharing for multi-way models, massively multilingual models, training protocols, language divergence, transfer learning, zero-shot/zero-resource learning, pivoting, multilingual pre-training and multi-source translation.

## tutorial overview the tutorial will draw material from a survey paper on multilingual nmt that we have authored @xcite . this paper has been published in acm computing surveys. we also intend to cover some latest advances that are not mentioned in the survey paper. we will divide the tutorial in two parts, the first focusing on general purpose multilingual modeling and the second focusing on multiple usecases for multilingual nmt. in the first part, we will first present an overview of the basics of nmt and cross-lingual embeddings. we will establish the fundamentals of mnmt and focus on various design choices. design choices will involve network architecture, training protocols, data processing, hyper-parameter tuning so that they can successfully incorporate multilingualism. we will discuss specific changes to be made for translation between related languages as well as unrelated languages. we will also talk about the limits of massively mnmt models and provide a cost-benefit analysis from the perspective of deploying such models. in the latter half of the tutorial, we will first focus on the challenging scenario of translation between language pairs for which there are few or no parallel corpora. we will introduce various ways to leverage pivot languages and on transfer learning approaches. we will show how transfer learning approaches such as fine-tuning and teacher-student learning can be optimally done when language relatedness and syntax are explicitly addressed. we will also touch upon unsupervised nmt which addresses low-resource mt using just monolingual corpora and is complimentary to nmt. we will see how multilingualism and unsupervised approaches can be combined. we will see how mt models for new languages can be rapidly adapted from pre-trained mnmt models. additionally, we will spend some time on multi-source nmt which leverages multilingual redundancy in terms of input in order to yield high quality translations. we will end the tutorial with a discussion on possible future directions that we believe that mnmt research should take. ## tutorial outline some representative papers are mentioned against tutorial sections.