Linguistically Motivated Subwords for E nglish- T amil Translation: U niversity of G roningen’s Submission to WMT -2020 (2020)

Abstract: This paper describes our submission for the English-Tamil news translation task of WMT-2020. The various techniques and Neural Machine Translation (NMT) models used by our team are presented and discussed, including back-translation, fine-tuning and word dropout. Additionally, our experiments show that using a linguistically motivated subword segmentation technique (Ataman et al., 2017) does not consistently outperform the more widely used, non-linguistically motivated SentencePiece algorithm (Kudo and Richardson, 2018), despite the agglutinative nature of Tamil morphology.

## introduction in this paper we present the neural machine translation (nmt) systems submitted to the wmt-2020 english-tamil (@xmath0. this task is challenging mainly for two reasons: through our submission we wish to provide solutions to the following questions: we start our paper with a short description of the tamil language before delving into the various techniques adopted by our submitted nmt systems. ## tamil language tamil is a dravidian language spoken by around 80 million people. tamil morphology is agglutinative and suffixal, i.e words are formed by suffixing morphemes to a lemma @xcite ). tamil suffixes can be either derivational (marking a change in pos and/or meaning) or inflectional. in particular, nouns in tamil are inflected for number, gender, case and animacy while verbs are inflected for tense, mood, aspect, negation, interrogation, information about emphasis, speaker perspective, sentience or rationality, and conditional and causal relations. all the aforementioned statements substantiate the fact that tamil morphology is highly complex. in fact, @xcite identified 716 inflectional rules for nouns and 519 rules for verbs. furthermore, designing a translation system for tamil is challenging given the lack of training data (compare the sizes of japanese and tamil parallel datasets in wmt 2020, both agglutinative, however having vastly different training data; 25m sentences and 630k, respectively). ## previous work one of the earliest automatic translation systems for @xmath1. they created a hybrid statistical/rule-based machine translation (smt) system and trained it on only 5k en-ta parallel sentences. @xcite created smt systems (phrase-based and hierarchical) that were trained on a much larger dataset of 190k parallel sentences. they also performed pre-processing steps involving morphological rules based on tamil suffixes that improved upon the bleu score of the baseline model (from 9.42 to 9.77). their dataset (henceforth called ufal) became the default benchmark for en-ta translation systems until 2019, and we also use it in our experiments as an additional (general-domain) development set. to the best of our knowledge, there have only been a handful of nmt systems trained on @xmath2. for the indic languages multilingual tasks of @xcite , dabre et al. (2018) and ojha et al. (2018) reported bleu scores for @xmath3. the phrasal-based smt system of ojha et al. (2018) with a score of 30.53 bleu outperformed the nmt systems of sen et al. (2018) (11.88) and dabre et al. (2018) (18.60), suggesting that the nmt systems were not suitable for translating a highly morphological language such as tamil. however, the following year, philip et al. (2019) outperformed ramasamy et al. (2012) on the ufal dataset with a bleu score of 13.05. they report that techniques such as domain adaptation and back-translation can make training nmt systems on low-resource languages possible. ## datasets for our constrained systems, we restrict ourselves to the datasets provided by wmt. parallel monolingual ## experimental setup all our nmts are developed using fairseq @xcite . following the architecture setup of @xcite the transformer-base implementation (base) is used, with slight changes to a few parameters, which are explained below. the encoder and decoder are both set to 5 layers with embedding dimension of 512 and 8 attention heads. the hidden layer dimension is 2048 and layer normalization is applied before each encoder and decoder layer. other parameters were set as follows: dropout (0.001), weight decay (0.2) and batch size of 4k tokens. our loss function is cross-entropy with label smoothing of 0.2. the model is trained for 100 epochs with early stopping criterion set to 3. ## results we report bleu scores on three testsets: the ufal testset @xcite , half of the wmt2020 devset (dev) foot_0 and the official wmt2020 testset. given the rich morphology of tamil, we also report chrf scores @xcite on the wmt2020 testset. we ran the program chrf++.py foot_1 with the arguments -nw 0 -b 3 to obtain the chrf score. from prior experimentation we found that a jointly trained sp model resulted in better bleu when compared to separate training for each language, and hence perform the majority of sp experiments in the last two contrastive experiments (exp8.2 and exp11.2) were run after the evaluation phase to better assess the impact of lmvr on translation quality in our best systems. the following observations can be made based on the results: differences across testsets the trends are often inconsistent across testsets. exp2 gave the highest bleu score on ufal (11.8) but a low bleu score for dev and wmt. on the other side, exp11 (and exp11.2) provided us the highest bleu score on the official wmt testset, but a low 10.5 for ufal. these variations could be attributed to the nature of the testsets and our training regime. because we focused on improving our nmt systems to adapt to the news genre of wmt testset, this resulted in loss of translation accuracy of the ufal testset, which was a mixture of three domains (one of them being news). ## analysis we also performed two small qualitative studies on the best systems based on segmentation. first, we compare how the segmentation algorithms segment the tamil word தக 'book'. secondly, using the example of the word book we observe how the systems translate the word to and from tamil ( segmentation translation quality for the compound comicbook, the sp system translates it as நைக ைவ /nakaiccuvai puk, i.e. comedy book, with the word /puk being a direct transliteration for book and hence incorrect. on the other hand, lmvr provides the correct translation. there were in total two occurrences of the nominative form of the தக /puththagam, which were correctly translated by the two systems. the same was observed for the locative form த-க ல/puththagamla. an example where both systems fail to translate the phrase by the book as in the sentence "i have every reason to believe they have done everything by the book and ...". the sp system provides finally we observed with the word notebook that sp generated a non-existent word and lmvr provided an another translation for the english word notebook. in the future, we aim to conduct in-depth analysis on what and which morphological features are captured by the nmt systems. ## conclusion although our results were not competitive with the other submissions for the en-ta task, our paper presents the various settings that leads to an improvement in en-ta translation. mainly, we found that linguistically motivated subword segmentation @xcite , which was previously shown to benefit translation from/into various non-indian languages, does not consistently outperform the widely used sentencepiece segmentation despite the agglutinative nature of tamil morphology. we also found that, for our english-tamil systems, fully-fledged back-translation remains more competitive than its cheaper alternative @xcite . and finally, we observe a noticeable chrf gain when adding word dropout @xcite to our best model.