Optimal Word Segmentation for Neural Machine Translation into D ravidian Languages (2021)

Abstract: Dravidian languages, such as Kannada and Tamil, are notoriously difficult to translate by state-of-the-art neural models. This stems from the fact that these languages are morphologically very rich as well as being low-resourced. In this paper, we focus on subword segmentation and evaluate Linguistically Motivated Vocabulary Reduction (LMVR) against the more commonly used SentencePiece (SP) for the task of translating from English into four different Dravidian languages. Additionally we investigate the optimal subword vocabulary size for each language. We find that SP is the overall best choice for segmentation, and that larger dictionary sizes lead to higher translation quality.

## introduction dravidian languages are an important family of languages spoken by about 250 million of people primarily located in southern india and sri lanka @xcite . kannada (kn), malayalam (ma), tamil (ta) and telugu (te) are the four most spoken dravidian languages with approximately 47, 34, 71 and 79 million native speakers, respectively. together, they account for 93% of all dravidian language speakers. while kannada, malayalam and tamil are classified as south dravidian languages, telugu is a part of south-central dravidian languages. all four languages are sov (subject-object-verb) languages with free word order. they are highly agglutinative and inflectionally rich languages. additionally, each language has a different writing system. the highly complex morphology of the dravidian languages under study is illustrated if we compare translated sentence pairs. the analysis of our parallel datasets (section 4.1, in this paper we describe our work on neural machine translation (nmt) from english into the dravidian languages kannada, malayalam, tamil and telugu. we investigated the optimal translation settings for the pairs and in particular looked at the effect of word segmentation. the aim of the paper is to answer the following research questions: in what follows, we review the relevant previous work (sect. 2), introduce the two segmenters (sect. 3), describe the experimental setup (sect. 4), and present our answers to the above research questions (sect. 5). ## subword segmentation techniques prior to the emergence of subword segmenters, translation systems were plagued with the issue of out-of-vocabulary (oov) tokens. this was particularly an issue for translations involving agglutinative languages such as turkish @xcite or @xcite . various segmentation algorithms were brought forward to circumvent this issue and in turn, improve translation quality. perhaps the most widely used algorithm in nmt to date is the language-agnostic byte pair encoding (bpe) by @xcite . initially proposed by @xcite , bpe was repurposed by @xcite for the task of subword segmentation, and is based on a simple principle whereby pairs of character sequences that are frequently observed in a corpus get merged iteratively until a predetermined dictionary size is attained. in this paper we use a popular implementation of bpe, called sentencepiece (sp) @xcite . while purely statistical algorithms are able to segment any token into smaller segments, there is no guarantee that the generated tokens will be linguistically sensible. unsupervised morphological induction is a rich area of research that also aims at learning a segmentation from data, but in a linguistically motivated way. the most well-known example is morphessor with its different variants @xcite @xcite . an important obstacle to applying morfessor to the task of nmt is the lack of a mechanism to determine the dictionary size. to address this, @xcite proposed a modification of morfessor flatcat @xcite , called linguistically motivated vocabulary reduction (lmvr). specifically, lmvr imposes an extra condition on the cost function of morfessor flatcat so as to favour vocabularies of the desired size. in a comparison of lmvr to bpe, @xcite reported a +2.3 bleu improvement on the english-turkish translation task of wmt18. given the encouraging results reported on the agglutinative turkish language, we hypothesise that translation into dravidian languages may also benefit from a linguistically motivated segmenter, and evaluate lmvr against sp across varying vocabulary sizes. ## results the nmt systems are evaluated and tested on the official development and test sets, respectively from wat21. these evaluation sets are sourced from the pmindia dataset @xcite . during validation, models are evaluated by bleu on the segmented data, whereas final test scores are computed on the un-segmented and detokenized sentences (de-tokenization is performed with the indic nlp library tool). in addition to bleu @xcite , we also report on chrf score @xcite , which is based on character n-grams and is therefore more suitable to assess translation quality in morphologically complex languages. 4 we report the macro-averaged document level f3-score.results are presented in sp clear winner for kannada and tamil: sp presented the highest bleu and chrf scores for kannada and tamil. when we compare the best systems for both sp and lmvr, large differences are observed. for kannada differences of +6 bleu and +7.4 are observed and for tamil the difm-popovic/chrf ferences are +14.9 for bleu and +5.9 for chrf. ## analysis 6.1 different subtokens generated ## conclusion we presented our work on neural machine translation from english into four dravidian languages (kannada, malayalam, tamil and telugu). several experiments were carried out to find out whether a linguistically motivated subword segmenter (lmvr) is more suitable than a purely statistical one (sentencepiece) for translating into the morphologically complex dravidian languages, while using a transformer architecture. while bleu results were mixed on malayalam and telugu, chrf scores clearly suggest that sentence-piece remains the best option for all of our tested language pairs. we also found interesting differences among the four target languages. though they all belong to the same language family and share various linguistic phenomena, they are different with respect to source/target token ratio ( finally, we invite future researchers to carry out research on dravidian languages, especially kannada. compared to the plethora of work found for other languages, the work on dravidian languages is lagging behind. as our results show, there remains a large space for improvements, particularly