The TALP - UPC System Description for WMT 20 News Translation Task: Multilingual Adaptation for Low Resource MT (2020)

Abstract: In this article, we describe the TALP-UPC participation in the WMT20 news translation shared task for Tamil-English. Given the low amount of parallel training data, we resort to adapt the task to a multilingual system to benefit from the positive transfer from high resource languages. We use iterative backtranslation to fine-tune the system and benefit from the monolingual data available. In order to measure the effectivity of such methods, we compare our results to a bilingual baseline system.

## introduction modern nmt systems such as transformer require large amounts of training data in order to obtain good generation results. for this reason, low resource languages represent a good opportunity to explore new techniques to treat data more efficiently and benefit from available sources of data like monolingual corpora. from the wmt20 news tasks proposed languages we are presenting our results on the english-tamil language pair, tamil is an official language from india, sri lanka, and singapore having approximately 75 million native speakers. it belongs to the dravidian family, originated in asia. two principal reasons can make tamil a challenging language for machine translation: script and agglutination. tamil's script consists of 12 vowels and 18 consonants plus one special character, allowing the combination of 247 possible characters. compared to the latin script employed by most western languages, it is an order of magnitude higher in the number of possible characters. also, by agglutination, suffixes can be added to root words to form new ones. these words can lead to multiple words in the target language in the context of machine translation, which may affect attention and decoding in nmt systems. this work discusses the system proposed for the evaluation in which we combine the use of multilingual parallel data with monolingual data to boost the performance of our proposed nmt system. ## low resource nmt modern nmt systems benefit from having hundreds of thousands or even millions of parallel sentences. when working with low resource language pairs, the two main approaches are the use of monolingual corpora and multilingual nmt. while parallel data may be difficult to obtain for low resource languages, monolingual data is usually more available, as it does not require any additional labeling. a common approach to benefit from monolingual data is back-translation @xcite , which consists of translating a monolingual corpus to generate synthetic corpora that can be later employed to continue training. similar techniques create a synthetic pseudo-parallel corpus through a pivot language @xcite that can be then trained similarly to back-translation when data is available between the desired language pair and a pivot high resource language. more recently, iterative back-translation @xcite was proposed. this technique allows the system to generate synthetic data while updating the system, so better the new data improves as the system trains. on the other hand, several works on multilingual nmt have shown benefits for low resource language pairs by allowing positive transfer from the high resource languages, boosting the performance of the low resource ones. different architectures have been proposed that show this behavior, from universal models where all parameters are shared between all languages @xcite , to architectures that share a common device that maps representations into a shared represen-tation space @xcite , to architectures that do not share parameters @xcite @xcite . in the context of the wmt20 tamil-english news shared task, as the provided parallel data is limited, we resorted to a combination of both proposed methods by incrementally train the new language pair into a multilingual nmt system using the provided parallel data, to later fine-tune the system using iterative-back-translation with monolingual corpora. ## related work previous works @xcite have shown that indian languages are usually a challenge for nmt systems due to their difference in terms of vocabulary and grammar compared to western languages such as english. also, standard preprocessing methods do not always work well with them, so specific solutions are required to obtain good results. in the context of nmt, previous systems, such as midas @xcite , proved that the use of subword units leads to significant improvements in translation quality when applied to tamil by preventing out of vocabulary words in at generation time. ## corpora and data preparation all proposed systems in this work are constrained using exclusively data provided by the task's organization. the multilingual initial system was trained using europarl v8, for all translation directions between english, french, spanish, and german. for english-tamil pmindia, tanzil v1, the ufal entam corpus, the nlpc uom en-ta corpus, wikimatrix, and wikitiles. as monolingual tamil data, we used news crawl, while for english, we used news-commentary. we processed all non-tamil data following moses @xcite scripts provided by the organization. for each one, we applied punctuation normalization, tokenization, and true-casing. then each language is independently tokenized using bpe @xcite with 32 thousand operations. as test set, we used 1275 lines extracted from the development set provided from the organization, keeping the remaining ones as validation set. ## system description in this section, we are going to discuss the details of the pipeline followed to create the translations systems for this submission, including the multilingual supervised pretraining and the unsupervised fine-tuning using monolingual corpora. ## experiments and results the motivation for this work was to explore the combination of both positive transfer and monolingual data in a low resource task such as english-tamil translation. to test our hypothesis we trained a bilingual baseline with just the parallel data available for the task and compared its results to an incremental using adaptation to a multilingual nmt system and monolingual fine-tuning to measure the impact of each measure in the final performance. all configurations have the same architecture and number of parameters and have been tested on the same 1275 lines extracted from the newsdev2020 tamil-english set. to introduce some context about the multilingual system, we evaluated its performance using newstest13 as test set, and the performance english performance ranged from 20.31 bleu points from the english-german translations direction, to 29.74 for english-french. when english is the target language the results vary from 24.54 for german-english, to 27.75 for spanish-english. about the impact of positive transfer from multilingual nmt, tables 4 and 3 show that both directions benefit from adding tamil into the mnmt system with improvement of 1.58 and 4.09 bleu points respectively, approximately a 40% better than the bilingual baseline in both directions. when looking at the monolingual fine-tuning results, we can observe that the english to tamil translation direction benefits more (2.65 bleu) from the technique than the tamil to english direction (1.02 bleu). this difference in the performance may be explained by the difference in the training of both decoders. while the tamil de-coder has been trained just with the parallel data for the task, the english decoder was trained with the multilingual nmt system with more data available, which may lead to a more robust model to fine-tuning. finally, looking at the checkpoint averaging results, in both directions it leads to a small improvement, less than 0.2 bleu, showing limited impact in the final results. ## conclusions in this paper, we described the talp-upc participation in the wmt20 news translation shared task for tamil-english. the motivation of this work was to explore the combination of multilingual transfer from high resource languages and monolingual data applied to low resource nmt. our experiments showcase the effectiveness of adapting low resource languages pre-trained multilingual systems and how it introduces positive transfer compared to a bilingual baseline system. also it shows that monolingual data can be successfully introduced into the system and that it can boost the performance of the system. as future work, we could explore the fine-tuning of both encoder and decoder during the monolingual unsupervised finetuning in order to help the system produce better synthetic data as the training takes place.