Bridging P hilippine Languages With Multilingual Neural Machine Translation (2020)

Abstract: The Philippines is home to more than 150 languages that is considered to be low-resourced even on its major languages. This results into a lack of pursuit in developing a translation system for the underrepresented languages. To simplify the process of developing translation system for multiple languages, and to aid in improving the translation quality of zero to low-resource languages, multilingual NMT became an active area of research. However, existing works in multilingual NMT disregards the analysis of a multilingual model on a closely related and low-resource language group in the context of pivot-based translation and zero-shot translation. In this paper, we benchmarked translation for several Philippine Languages, provided an analysis of a multilingual NMT system for morphologically rich and low-resource languages in terms of its effectiveness in translating zero-resource languages with zero-shot translations. To further evaluate the capability of the multilingual NMT model in translating unseen language pairs in training, we tested the model to translate between Tagalog and Cebuano and compared its performance with a simple NMT model that is directly trained on a parallel Tagalog and Cebuano data in which we showed that zero-shot translation outperforms a directly trained model in some instances, while utilizing English as a pivot language in translating outperform both approaches.

## introduction neural machine translation (nmt) is an approach that trains a single neural network to translate between two languages using a parallel dataset. in a recent survey paper of @xcite on low-resource nmt, it has been shown that most of the recent works is highly data-inefficient and its performance drops sharply on low-resource set-tings. several techniques for addressing problems of nmt in low-resource settings were also discussed, the most recent of which includes exploiting multilingual resources through multilingual nmt, an extension of the traditional nmt that translates across multiple languages. google's approach in developing a multilingual translation system was discussed by @xcite , the research was able to highlight how combining highresource languages with low-resource languages helps in improving the latter's translation quality as compared to a standalone model. zero-shot translation is a surprising benefit of their work that bridges translation between unseen language pairs. on a practical setting, multilingual nmt is a more efficient approach when dealing with multiple translation scenarios as compared to having a single model per language pair as it is easier to deploy and maintain it in production. an interesting area of study in multilingual nmt has recently been explored by @xcite in which they studied the effect of clustering similar languages in a multilingual nmt, they measured the similarity of languages in two ways: prior knowledge (i.e. geographical proximity, language family, topology) and through the learned language embeddings. they have shown that clustering similar languages yields better translation results as compared to randomly clustered languages. one thing that was not able to investigate by @xcite is the effectiveness of a many-to-many multilingual nmt and zero-shot translation on closely related languages (e.g. philippine language family). the philippines is home to 186 languages based on (eberhard and gary). 184 of these are considered to be living and 2 are already extinct. 175 of the living languages are indigenous and 9 are non-indigenous. additionally, 37 in the living are institutional, 68 are developing, 38 are vigorous, 30 are in trouble, and there are 11 dying languages. there are also 3 unestablished languages. according to @xcite , the philippine languages may be grouped into language families, which includes the northern group (ilokano, pangasinan, and kapampangan), and a central group (tagalog, bicolano, hiligaynon, and cebuano). @xcite also mentioned that the philippines is experiencing a period of language convergence, a type of linguistic change in which languages structurally resembles or blends into one another as a result of prolonged interaction, manifested by high volume of borrowing from large languages such as english, tagalog, and other regionally significant languages. philippine languages also has large amount of intersections and sharing across the languages according to the work of @xcite on classifying low-resource philippine languages. in recent years, there has been a number of research works on developing neural machine translation systems but all of them are english-centric and are focusing on tagalog @xcite and cebuano (adlaon and marcos, 2018). machine translation systems exists for other philippine languages according to the survey paper of @xcite but implemented on either statistical, transfer-based, or corpus-based approach. to the best of our knowledge, only the work of (adlaon and marcos, 2018) has developed and investigated translation between two philippine languages where they developed a unidirectional translation system for cebuano to tagalog. ## related works in recent years, neural machine translation has been extended to support multilingual translation which has been strongly motivated by the following reasons: a) maintaining a single model for each language pair or translation direction is challenging in an industry setting @xcite ; and b) exploiting multilingual data in improving lowresource translation settings @xcite . in one of the earliest works in multilingual nmt, @xcite extended the traditional nmt into a multitask learning framework by implementing a shared encoder for the source language and dedicated encoders for the target languages, this resulted in a translation performance that achieves higher bleu scores than individually trained models. on the other hand @xcite mod-ified the traditional nmt by implementing multiple encoders and decoders sharing a single attention mechanism and also shown improvements on low resource translation performance. extending the traditional nmt systems to handle multilingual translation would typically require some major changes in the architecture as shown in the works of @xcite and @xcite which is something that is addressed by the work of @xcite where their implementation of multilingual nmt does not require major changes in the architecture but only requires to add a special token in the dataset to signify the translation direction. @xcite conducted the experiments in three different settings: one to many (e.g. @xmath0,portuguese), many to one (e.g. spanish,@xmath1, and many to many (e.g. english ↔ spanish, english ↔ portuguese), the most interesting experiment is the many to many setting where an evidence of transfer learning occurred when the model learned to do zero-shot translation or to translate between unseen language pairs. in zero-shot translation, since the model was trained to translate between @xmath2, the model can generate acceptable translations for @xmath3. more recently, the work of @xcite implemented a transformer-based multilingual nmt based on the works of @xcite , the main contribution of their work is the language clustering in building a multilingual nmt that aims to answer the question of which languages should be grouped together in order to gain an optimal result. @xcite based the clustering on two methods, using prior knowledge (e.g. language family, geographical proximity) and language embedding, their work shows that clustering languages yields better results in almost all of the translation tasks as compared to randomly grouped or individually trained models. however, the work of @xcite did not conduct a many to many experiment which would have been interesting given that it enables zero-shot learning and it is intriguing to see the effect of zero-shot learning on closely related language pairs. ## data a parallel corpora sourced from the opus project @xcite consisting 6 language pairs was used to train two of our models. in an attempt to investigate on the relation or effect of clustering multiple languages to machine translation performance, we visualized the embeddings of selected instances in while perplexity is a tuneable hyper-parameter that is used to estimate the number of close neighbors a point has, figure 1 shows that in all of the perplexity configurations, 5 observable clusters were formed. we zoomed-in to the clusters and isolated the points belonging to each group as seen in figure 1 . it was observed that articles the and a are grouped in clusters 1 and 4, respectively, while symbols . (period) and , (comma) in clusters 2 and 5, respectively. while words like classmates and study were found clustered together in cluster 3. clustering in figure 1 shows that both semantic and syntactic relations of words in multilingual pairs could be captured but is highly evident for the english language (as shown in figure 1 ) as it is the language present in all of the pairs. with a goal of being able to aid the automatic translation of mother-tongue based multilingual education (mtb-mle) materials used by the department of education in the philippines; a separate work was conducted gathering tagalog-cebuano parallel dataset in the academic domain. we continued to gather and build an academic domainspecific parallel corpus for tagalog and cebuano languages as initiated by the work of (adlaon and marcos, 2019). due to the absence of parallel data in this domain, we manually identified specific articles and categories in wikipedia to be extracted. categories that were identified should be somehow related to the mtb-mle topics. wikipedia as the dataset source, has aligned article pairs that may range from being almost completely parallel to containing almost no parallel sentences. ## methodology there are four fundamental parts in the methodology, the collection of parallel data from various sources, preparation of the data, model development and training, and the testing and evaluation of the models. two models were trained and developed on the model training phase, a multilingual nmt model trained on parallel english and philippine languages data, and a direct nmt model trained on a parallel tagalog and cebuano data. consequently, two types of model testing was conducted, direct and comparative evaluations. the direct evaluation includes the direct evaluation of the multilingual nmt model on the english and philippine languages parallel test data, and evaluation of its capability to translate unseen language pairs from training through the zero shot and pivot translation using the english language on parallel data of various philippine languages. lastly, the comparative evaluation includes the performance comparison of a model directly trained on a tagalog and cebuano parallel data against a multilingual nmt model that has not seen any tagalog and cebuano parallel data in training, relying only on its capability to translate with zero shot translation and pivot translation. ## experimental setup all experiments were conducted using the transformer architecture of @xcite with relevant modifications for each model, and has been evaluated with the bleu score based on @xcite . bleu is the averaged percentage of n-gram matches @xcite that for each i-gram in @xmath4, 2, ..., n , the percentage of the igram tuples in the hypothesis that also occur in the references is computed: ## results and discussion we used two approaches in multilingual translation from the mnmt model; zero-shot and pivot-based approach. the performance of these approaches were then compared to dnmt model which was trained on a tagalog-cebuano bilingual corpus. for the multilingual models, translations were detokenized before scoring. translation results were then evaluated using sacrebleu, a python script that computes for bleu scores except that it expects detokenized outputs @xcite from models. ## conclusion in this paper, we presented a benchmark work on multilingual neural machine translation for major philippine languages. we presented two translation approaches from the model that seek to address the problem of low-resource machine translation using zero-shot and pivot-based techniques. results show that the pivot-based technique provides the closest translation compared to zero-shot and dnmt. although zero-shot may be a good technique for mnmt, the translations it generated oftentimes contain code-switching mostly english words as discussed in the lexico-syntactic divergence section. compared to zero-shot and pivot-based techniques, dnmt handled the retention of the sentence structure well. this could be attributed to the model's preprocessing mechanism. we also could not see any relation to the result of our attempt to cluster languages in machine translation performance, but we see this as a future direction of study. for future work, we plan to have a thorough extrinsic evaluation of the performance of the mnmt models for other ph language pairs other than tagalog and cebuano including further investigation of the clustering of embeddings as it relates to translation results. also, in-depth work will be conducted on addressing the issues identified in the evaluation. in parallel to improving the mnmt models, a thorough investigation of having both morphologicallycomplex language pairs (e.g. tagalog and cebuano) versus complex and non-complex language pairs as a dataset will be studied.