H indi to D ravidian Language Neural Machine Translation Systems (2023)

Abstract: Neural machine translation (NMT) has achieved state-of-art performance in high-resource language pairs, but the performance of NMT drops in low-resource conditions. Morphologically rich languages are yet another challenge in NMT. The common strategy to handle this issue is to apply sub-word segmentation. In this work, we compare the morphologically inspired segmentation methods against the Byte Pair Encoding (BPE) in processing the input for building NMT systems for Hindi to Malayalam and Hindi to Tamil, where Hindi is an Indo-Aryan language and Malayalam and Tamil are south Dravidian languages. These two languages are low resource, morphologically rich and agglutinative. Malayalam is more agglutinative than Tamil. We show that for both the language pairs, the morphological segmentation algorithm out-performs BPE. We also present an elaborate analysis on translation outputs from both the NMT systems.

## introduction machine translation has improved extensively using deep neural networks with the utilization of large dataset and high computational capacities. the successful works in neural machine translation (nmt) started with the encoderdecoder based architecture presented by @xcite , @xcite . sutskever et al (2014) built nmt system using long short term memory (lstm) to overcome the fixed-length vector constraint in the previous architecture. bahdanu @xcite introduced the attention mechanism, where bidirectional recurrent neural network (rnn) consisting of forward and backward rnn was used to focus around the word. this attention mechanism was simplified by considering the hidden states at the top layer of both encoder and decoder by @xcite . transformer, an architecture where encoder and decoder completely relying on the attention machines was presented by @xcite . though these nmt systems have achieved a state-of-art performance in high-resource, closely related languages, its performance drop significantly in low-resource and morphologically rich languages. some of the techniques employed to mitigate challenges in handling the lowresource languages are as follows; increasing the data using back translation, utilisation of phrase tables generated in smt, leveraging the pretrained models, combining the similar language data and using transfer learning. the morphological rich languages are handled using different sub-word segmentation techniques, which helps in reducing the vocabulary size and increasing the number of examples of each tokens. in this work, we compare the morphologically inspired segmentation methods against the byte pair encoding (bpe) in processing the input for building nmt systems for hindi (hi) to malayalam (ml) and hindi to tamil (ta), where hindi is an indo-aryan language and malayalam and tamil are south dravidian languages. these two languages are low-resource, morphologically rich and agglutinative. further the paper is organised as follows. in the following section, we present a summary on the different sub-word tokenisation works in nmt. this is followed by details on related works in indian language nmt. in the third section, we describe briefly the characteristics of three languages, which highlight the challenges in building nmt systems for hindi to malayalam and tamil. in section 4, we describe our experimental setup and data preparation. the result and analysis is presented in section 5. we conclude the paper with a conclusion section containing the gist of the work. ## related works the common strategy of handling the morphologically rich languages in nmt is to apply sub-word segmentation. this reduces the vocabulary size and increases the frequency of the tokens and improves the translation by handling rare words and unknown words, but it introduces grammatical errors. @xcite presented the different word segmentation techniques which included simple character ngram model and segmentation based on the byte pair encoding (bpe) comparison algorithm. bpe sub-word algorithm is one of the widely used subword tokenisation algorithm. the other sub-word tokenisation algorithms include, wordpiece, sentencepiece, mecab (a morphological analysed based japanese tokeniser), stanford word segmentation ( a chinese word segmentor based on conditional random fields), opennmt tokenizer and moses tokenizer (normalise characters and separates punctuation from words). there are various attempts in modifying the existing tokenization techniques and few are listed here. @xcite extended the bpe segmentation by including two other statistical measures namely accessor variety (av) and description length gain (dlg). they evaluated it with german to english and chinese to english translation. provilkov et. al. (2019) introduced bpedropout, where segmentation procedure of bpe was stochastically altered to produce multiple segmentations within the same fixed bpe framework. wang et. al. (2020) focussed on byte-level bpe (bbpe), where the text is tokenised into variable-length byte n-grams instead of character level sub-words. nonaka et. al. (2022) has presented a locally consistent parsing (lcp) stochastic string algorithm to achieve optimum compression instead of bpe compression, which has the drawback in generating multiple segments. tang et. al. (2020) performed a study on pure character based model in translating finnish to english. they have demonstrated that the word level information is distributed over the entire character sequence and character at different position play different roles in learning linguistic knowledge. deguchi et. al. (2020) performed tokenisation of sentences by using sub-word units induced from bilingual sentences. here the tokenisation of sentences is performed by considering its translation. nguyen et. al. (2020) proposed an approach, where the heterogeneous translation units were used to build in russian to vietnamese nmt. they considered linguistic characteristics of syntactic russian and analytic vietnamese. machacek et. al. (2019) compared the linguistically motivated method morfessor and derivational dictionaries based method and statistical methods such as ste and bpe in german to czech translation. their experiments showed the non-linguistically motivated method performed better. in this sub section, we present a gist of the nmt works published in indian languages. @xcite has presented hindi to english nmt, where they generalised the embedding layer of the transformer model to incorporate linguistic features such as pos, lemma, and morphological features. there was a significant increase in the bleu scores. @xcite has presented an elaborate nmt experiments to understand the poor performance of the dravidian languages compared to indo-aryan languages. they used byte pair encoding (bpe) method to understand the bpe in indian languages. from their study, they presented that the optimal value for bpe merge for indian languages is between 0-5000, which is low compared to that observed for european languages. wmt21 had a similar language task, which has boosted the research to explore the use of shared vocabulary in nmt. @xcite and @xcite has presented their work in tamil-telugu translation. @xcite has presented their work in marathi-hindi bidirectional translation. in the next section, we present a brief note on the characteristics of the languages considered. ## resources characteristics of the languages as mentioned earlier, in this work, we describe the nmt experiments in hindi to malayalam and hindi to tamil translation, where hindi is an indo-aryan language and malayalam and tamil are dravidian languages. the three languages are similar in the following features: verb final, relatively free word order, morphologically rich in inflections. and these languages are dissimilar in agglutination. malayalam and tamil have agglutination and hindi does not have. malayalam has more agglutination than tamil. the other differences are as follows. hindi and tamil have number, gender and person agreement, whereas malayalam does not have. hindi is an ergative language. in the ergative constructions, finite verb has agreement with the object. malayalam and tamil are nominative-accusative languages. malayalam and tamil have distinctive case markers, whereas in hindi, case marker 'se' occurs as instrumental, accusative and ablative case marker. this leads to one to many in case mapping between hindi to malayalam and tamil. in hindi, plural marker is affixed to the noun and case markers are written separately. in the case of pronouns, case markers are also affixed to the pronouns. in malayalam and tamil, both plural markers and case markers are affixed to the nouns. copula verb is obligatory in hindi and malayalam whereas in tamil it can be dropped. malayalam and tamil has distinctive 3 rd person pronouns (avan, aval, avar, athu), whereas in hindi, 'vaha' is used for all 3 rd person singular pronouns. the clausal construction in hindi varies with malayalam and tamil. in hindi, the clausal constructions are introduced by relativecorrelatives such as (jo-vo, agar-tho, jisa-usa, jisne-usne, jab-thab etc). in malayalam and tamil, the clausal constructions are introduced by nonfinite verbs namely, relative participle verb, conditional, infinite verb and verbal participle verb. it is further explained with the following example 1. ex 1: hi: agar barish ayege tho paani rain(n) come(v)+future water(n) milegaa. get(v)+future here 'agar' and 'tho' are the relative-correlative ml: mazha peythaal, vellam rain(n) rain(v)+cond water(n) labikkum. get(v)+future ta: mazhai peythaal, thanneer rain(n) rain(v)+cond water(n) kidaikkum. get(v)+future (if it rain, we will get water.) in the above example 1, conditional sentence is presented in hindi, malayalam and tamil. in hindi the conditional clause is introduced with the relative-correlative 'agar-tho', whereas in malayalam and tamil it is introduced by the nonfinite verb using the suffix '-aal'. negation in verb phrase in hindi varies with malayalam and tamil. in hindi, the negation occurs before the finite verb and in malayalam and tamil, it occurs as an auxiliary verb. consider the following example 2. ex 2: hi: vaha nahi aaya. he(pn) not(neg) come(v)+past+3sc ml: avan vannilla he(pn) come(v)+inf+aux (neg) ta: avan varavillai (vara+illai) he(pn) come(v)+inf+aux (neg) in example 2, the difference in construction of negation verb in hindi and malayalam and tamil is clearly seen with the position of the negation. these variations between hindi and malayalam and tamil in clausal structure, case markers, pronouns and verb construction introduce challenge in hindi to dravidian language translation. in the next section, we describe the corpus and the experimental setup. ## experiment in this section, we discuss about the details of the parallel dataset, experimental setup for developing hindi to malayalam and hindi to tamil nmt systems and data preparation for three different experiments. ## results and analysis we evaluated the translations from the three nmt models for both hindi to malayalam and hindi to tamil using bleu score @xcite . we used sacre-bleu python library to calculate the bleu scores. the results are presented in table (3). the bleu scores show that the morphological segmentation has significantly improved the translation in both hindi-malayalam and hindi-tamil. on analysis of the translation output from the three different experiments in both hindi to malayalam and hindi to tamil, our observations are as follows, word-level: many named entities, technical words and verb phrases occurred as unknown word (<@xmath0. bpe: translated sentences were complete but most of these translations were not the exact translation. ## conclusion we have presented our experiments in building neural machine translation system for hindi to malayalam and hindi to tamil, where we compare the morphologically inspired segmentation methods against the byte pair encoding (bpe) in processing the input for building nmt systems. hindi is an indo-aryan language and malayalam and tamil are dravidian languages. all the three languages are morphologically rich language. malayalam and tamil have agglutination. we have briefly explained the characters of these languages. we have compared the translation output from the word-level (base line) system and nmt systems trained with these two different sub-word processed data. word-level system had unknown words and verb generation was not proper. bpe system translation outputs were complete sentences but these translations were not exact translation. the sense of the sentences varied from the source sentence. bpe system handled unknown words. it also had errors. translation from morph-seg systems had a significantly high bleu score. the sense of translated sentences was close to the source sentences. unknown words are a challenge in this system, but it is comparatively less than the word-level system.