Unsupervised Machine Translation On D ravidian Languages (2021)

Abstract: Unsupervised Neural Machine translation (UNMT) is beneficial especially for under-resourced languages such as from the Dravidian family. They learn to translate between the source and target, relying solely on only monolingual corpora. However, UNMT systems fail in scenarios that occur often when dealing with low resource languages. Recent works have achieved state-of-the-art results by adding auxiliary parallel data with similar languages. In this work, we focus on unsupervised translation between English and Kannada by using limited amounts of auxiliary data between English and other Dravidian languages. We show that transliteration is essential in unsupervised translation between Dravidian languages, as they do not share a common writing system. We explore several model architectures that use the auxiliary data in order to maximize knowledge sharing and enable UNMT for dissimilar language pairs. We show from our experiments it is crucial for Kannada and reference languages to be similar. Further, we propose a method to measure language similarity to choose the most beneficial reference languages.

## introduction around 200 million people in the world speak a dravidian language which has almost 80 varieties @xcite . given the significant number of speakers, improving translation quality on these languages is an important task to facilitate intercultural communication. while neural machine translation (nmt) systems @xcite led to great improvements on high resource languages, a major challenge remains when there is little parallel data between the source and target languages. compared to resource-rich languages like german and french, the amount of high-quality parallel data for dravidian languages is minimal. as a result, the current translation quality for dravidian languages is largely lagging behind. unsupervised neural machine translation (unmt; @xcite learns to translate between languages without relying on parallel data. instead, only monolingual corpora are required. this relieves the need to acquire large parallel datasets, which is especially difficult for low resource languages. despite this attractive property, existing literature @xcite suggests that unmt performs well under the following conditions: 1) source and target languages are similar; 2) monolingual data for both the languages belong to the same domain. in most realistic scenarios that call for unsupervised translation, these conditions are rarely fulfilled. considering these challenges, one promising research direction to address these conditions is to combine multilingual nmt models with unsupervised translation @xcite @xcite . incorporating similar high resource languages could alleviate the data scarcity of low resource languages by exploiting the language similarity between them. @xcite describes this scenario as "unsupervised machine translation with multilingual transfer". for example, @xcite investigate translating between german to upper sorbian with no parallel data between them but using parallel data between the high resource languages german and english. among dravidian languages, the comparatively better resourced ones are tamil, telugu, malayalam and kannada. in turn, among these four languages, kannada is relatively low resource compared to the others @xcite . therefore, we consider a realistic scenario of improving translation quality on kannada using auxiliary data in the other three languages. we refer to the additional language as reference language following the terminology in @xcite . in this work, with a focus on dravidian languages, we analyze the influence of related languages as auxiliary data in various setups of multilingual unmt (munmt). as shown in figure 1 , we focus on a scenario of translating between kannada and english with the following conditions: 1) no parallel data for kannada-english; 2) very limited in-domain parallel data (∼30k sentences) between english and three other dravidian languages; 3) monolingual data for all five languages in a different domain from the test data. our main contributions in this paper are: ## related work initial work on unmt @xcite demonstrates the possibility of building unsupervised bilingual systems between similar and high resource languages. however, as shown recently @xcite @xcite , purely unsupervised systems perform poorly in situations with low resource and distant language pairs . however, often it is in this very case that unsupervised translation could be helpful. @xcite propose a hybrid approach that translates between nepali and english without parallel data by utilizing parallel data between hindi and english. specifically, the model is trained jointly with supervised loss between english and hindi and unsupervised loss between english and nepali. in our work, we employ additional loss terms on top of this in order to more tightly connect the languages without parallel data. moreover, we analyze ways to incorporate auxiliary parallel data and investigate the correlation to language similarity. the current state-of-the-art unmt systems build upon multilingual pretraining @xcite and include a large number of languages from diverse languages families. the high degree of multilinguality is expected to lead to better model generalization. in this work, we zoom in on dravidian languages and study how we can best utilize the related languages within this particular family. we also study data scenarios with minimal auxiliary parallel data and use a training procedure that relies more on the monolingual data. the languages in our setup belong to the same family, but have unique scripts. because of this, there is little vocabulary overlap. chakravarthi et al. @xcite compare the impact of transliteration in supervised multilingual nmt and shows that it was helpful. different from chakravarthi et al. (2019c), the focus of our work is unsupervised translation. we study the role of lexical overlap in a unsupervised training pipeline, including pretraining on monolingual data and supervised training on auxiliary language pairs. we also highlight the information loss caused by transliteration from code-mixing in data from certain domains. ## background we define l p as the set of language pairs with parallel data, and l m as the set of languages with monolingual data. the sample space of sentences for language @xmath0. similarly, for language pair x -@xmath1, the sample space of parallel sentences is φ x-y . a standard bilingual supervised model translating from language x to y is trained on parallel dataset φ x-y . the model parameters θ are optimized to maximize the likelihood of target sen-tences in y given the source sentences in x: in the auto-encoding task, the model aims to reconstruct sentences from corrupted inputs. specifically, a noise model c(x) corrupts an input sentence x by dropping and shuffling the words according to a given probability. the loss function for auto-encoding using a monolingual dataset for language x from the collection l m is defined as in the on-the-fly back-translation task, the model only needs monolingual data and its loss function is defined as the full loss function for unsupervised translation combines the auto-encoding loss in equation (2) on x and y respectively and the backtranslation loss in equation (3) between x and y in both directions: cross translation @xcite is a recently proposed technique that uses a reference language to improve the performance of unsupervised multilingual models. the procedure is similar to on-the-fly back-translation but additionally utilizes parallel dataset with reference languages. when training an unsupervised translation model from x to y , we use an additional parallel dataset x -r from the collection l p , where r is the reference language. beside the unmt loss, we additionally optimize the following cross translation loss function: for both on-the-fly back-translation and cross translation, the quality of intermediate translations is essential for the final unmt model @xcite . high-quality intermediate translations further needs similar encoder outputs for parallel sentences in the source and target language. while the denoising auto-encoding objective is expected to encourage this, achieving this is expected to be easier between similar languages. therefore, in this work we focus on utilizing similar languages for unmt. ## language similarity in unsupervised machine translation as outlined in section 2, distant language pairs pose a great challenge to unmt systems. in order to apply unmt between highly dissimilar languages, in our case english and kannada, we explore several directions to leverage language similarity using other reference languages in the dravidian family. first, we investigate the role of transliteration in unmt, as the languages we work with do not share a common writing system. secondly, we evaluate multiple model architectures to enable unsupervised translation of dissimilar language pairs. we experiment with using auxiliary data from various related reference languages in each architecture to show the influence of language similarity. last but not least, we propose a method to measure language similarity using n-gram overlap. based on this metric, we show the benefit of including reference languages with high similarity score. ## experiments and results first, we analyze the influence of having a unified writing system based on the approach described in section 4.1. next, we evaluate multiple architectures with different reference languages described in section 4.2. we then discuss our results in the context of language similarity. ## conclusion we focus on the challenging yet realistic scenario of translating between english and kannada without any parallel data. after confirming the difficulty of purely unsupervised translation between distant language pairs, we show that increasing the lexical overlap by transliteration is necessary for achieving unmt for our focal language. moreover, we utilize monolingual and limited amount of auxiliary parallel data in languages related to kannada. the best performance is achieved by unmt between similar languages along with cross translation. we show the importance of incorporating languages with high relatedness to the focus language by evaluating different reference languages. finally, we propose a metric for measuring lexical overlap and show that it serves as a good indicator for selecting the best reference languages.