MALM : Mixing Augmented Language Modeling for Zero-Shot Machine Translation (2022)

Abstract: Large pre-trained language models have brought remarkable progress in NLP. Pre-training and Fine-tuning have given state-of-art performance across tasks in text processing. Data Augmentation techniques have also helped build state-of-art models on low or zero resource tasks. Many works in the past have attempted at learning a single massively multilingual machine translation model for zero-shot translation. Although those translation models are producing correct translations, the main challenge is those models are producing the wrong languages for zero-shot translation. This work and its results indicate that prompt conditioned large models do not suffer from off-target language errors i.e. errors arising due to translation to wrong languages. We empirically demonstrate the effectiveness of self-supervised pre-training and data augmentation for zero-shot multi-lingual machine translation.

## introduction machine translation is one of the classic problems in natural language processing(nlp). several products like google translate, bing translate provide services to millions of translation requests across a diversity of language pairs. while the requests for these services come in almost all language pairs imaginable, the quality of translation for low-resource language pairs like german-arabic is especially low. this is prompted due to a lack of quality training data for such locale pairs compared to high-resource languages like english-french etc. specifically, for few-shot machine translation, there have been many successful techniques proposed. @xcite demonstrated that transfer learning from high-resource languages to lowresource languages can be used to achieve remarkably high bleu scores. building on top of it, @xcite showed that universal lexical representations achieve better alignment of lexical and syntactic relations between languages. similarly, @xcite have been successfully used to utilize computer vision leanings in augmentation to low-resource translation. success due to techniques like transfer-learning, data augmentation, etc. has also provided great progress in building large multi-lingual neural machine translation models @xcite . the objective here is to build a single high-capacity model that is able to generate translations for any language pair and can be trained at the same time. @xcite have used conditional specific language routing for achieving impressive performance across low resource language pairs. similar to this work, @xcite utilized data augmentation strategies @xcite used random online back translation to achieve state-of-the-art performance for low-resource machine translation. in their work, @xcite encourage parameter sharing across language by implementing an auxiliary loss function. similarly, de-noising objective @xcite and distillation techniques @xcite have also been shown to have boosted zero translation learning. recently, research direction in massively multilingual translation models(mmt) @xcite has also been popularized to build zero-shot translation systems. @xcite provides a survey of challenges associated with mmt models, while also emphasizing the importance of preprocessing and vocabulary in knowledge transfer across language pairs. although, @xcite detail the lack of robustness of zero-shot models across training runs, we do not notice it in our training runs and find that augmentation techniques help stabilize the training process. neural models like transformer @xcite have brought significant advances in tasks across nlp. pre-trained language models like bert @xcite , bart @xcite , t5 @xcite have achieved stateof-art performance across the nlp spectrum. similarly, generative models like gpt-2 (brown et al., 2020) have shown few and zero-shot abilities on many tasks. the wide success across tasks has not only been limited to high-resource languages like english, and french but has indeed been shared with low-resource languages like azerbaijani, belarussian, galician, urdu, etc @xcite . it has also brought significant progress in single large multi-lingual models mbert @xcite , mbart @xcite , mt5 @xcite ) that learn universal representation across languages. this is responsible for remarkable zero and few-shot performance across tasks for languages that lack supervised training data. in this paper, we will investigate language modeling pre-training and data augmentation strategy for zero-shot translation our work provides 2 major and concise pieces of contributions: ## problem we will first look at the challenge of off-target translation. for all the zero-shot language pairs, we construct a random test dataset of 1000 examples foot_0 from the source language that may or may not be part of the original test dataset. we then run the translation system over those examples and identify all the output that corresponds to the wrong target language. say, a translation model(m ) generates for data instances(x 1 ...x n ) translations as (y 1 ...y n ) and reference translations as (y 1 ...y n ) for source language(s) and target language(t). also, given a language identification oracle as l, where in the original paper, zhang et al. ( 2020 ) used the accuracy of translation language as a metric to compare. we argue that any language identification system is noisy and thus accuracy on just translation output doesn't take into account errors of the language identification system. otter, on the other hand, is a noisy measure that measures language accuracy over both reference and translation output text of the translation system. the main question that we investigate is improving the quality of zero-resource translation. the problem at hand is learning a single model that is able to learn translation across language pairs that are unseen during the training time. this is motivated by human language learning experience, that if a person knows german, english, and arabic and can translate over @xmath0. this is true for us because beneath all the lexical and grammatical differences across languages, we share the grounding of various concepts in the same representation. basically, the representation of the concept 'cat' is the same as the word 'katze' in german. traditionally, a pivot language has been prominently used to achieve this task. for example; if a german-to-arabic translation is required then the original text is passed through the first germanto-english translation engine, and then the output english sentence is passed through the english-to-arabic translation system. this simple yet effective strategy has been shown to achieve state-of-art performance across zero-resource settings in many language pairs. currey and heafield (2019) use data augmentation with pivot language to generate pseudo-parallel data across zero-shot language pairs and then re-train a system. recently, @xcite have even utilized multi-pivot languages and simultaneous translation as a method to improve zero-shot performance. while @xcite combined pivoting with transfer learning and an adapter module, @xcite leveraged monolingual data with self-supervision for low resource languages to achieve impressive performance. ## experimental setup for the purpose of this study, we will constrain the study and experiments to opus-100 by @xcite . opus-100 is an english-centric dataset with over 100 language pairs that have either source/target language as english. it also consists of several other non-english-centric pairs that are available for zero-shot translation objectives. we should emphasize that while previous and related work on this dataset has been centered around massively-multilingual translation as well as zeroshot translation, the objective of the current work is only on zero-shot translation. as part of zero-shot languages, opus-100 provides 15 language pairs that are combinations of french, german, arabic, russian, chinese and dutch. for the evaluation of our translation model, we shall use the bleu score, which is a standard metric of automatic evaluation across machine translation. we run our experiments using the mt5 implementation available in the transformers library provided by huggingface @xcite and use pre-trained mt5 models(small, large, and xx-large) from @xcite . for reproducibility purposes, we use the adam optimizer and run our experiments on a google tpu v2-32 instance for 64,000 steps with 256 max length, 512 batch size, 0.0001 learning rate and we use a beam size of 4 at inference time. for baseline experiments, we consider 2 strong baselines -(i) first, for any language pair say xx-yy, we train 2 transformer models xx-en and en-yy, and run zero shot inference for any new sentence from language xx through xx-en and then through en-yy, note that we do not pre-train these models (ii) second, we consider the model from @xcite which implements random online back translation to recover from off-target translation. ## results & analysis we find that a mixed data augmentation training regime helps bring down otter to an extremely low range as referenced in 1. we attribute this to the compositional relationships learned by the large language model on the mixed vocabulary as well as the language tag we use as part of the task prompt. similarly, we find that data augmentation techniques like seq2mix @xcite , can substantially improve zero-shot performance when used on top of large pre-trained language models. we explain the performance using the following reasoning: ## conclusion in this paper, we utilized mixing augmentation techniques along with large sequence-to-sequence models to generate high-quality zero-shot translation models for language pairs that have no training data available. we successfully demonstrate that large pre-trained language models are able to learn the semantic spaces between languages and are already good at zero-shot machine translation. however, data augmentation techniques can further boost this performance to achieve impressive results on zeroshot translation.