NITK - U o H : T amil- T elugu Machine Translation Systems for the WMT 21 Similar Language Translation Task (2021)

Abstract: In this work, two Neural Machine Translation (NMT) systems have been developed and evaluated as part of the bidirectional Tamil-Telugu similar languages translation subtask in WMT21. The OpenNMT-py toolkit has been used to create quick prototypes of the systems, following which models have been trained on the training datasets containing the parallel corpus and finally the models have been evaluated on the dev datasets provided as part of the task. Both the systems have been trained on a DGX station with 4 -V100 GPUs. The first NMT system in this work is a Transformer based 6 layer encoder-decoder model, trained for 100000 training steps, whose configuration is similar to the one provided by OpenNMT-py and this is used to create a model for bidirectional translation. The second NMT system contains two unidirectional translation models with the same configuration as the first system, with the addition of utilizing Byte Pair Encoding (BPE) for subword tokenization through the pre-trained MultiBPEmb model. Based on the dev dataset evaluation metrics for both the systems, the first system i.e. the vanilla Transformer model has been submitted as the Primary system. Since there were no improvements in the metrics during training of the second system with BPE, it has been submitted as a contrastive system.

## introduction tamil is a language, predominantly spoken in tamil nadu, a state in southern india, along with countries with a large tamil speaking diaspora such as sri lanka, malaysia and singapore, to name a few. telugu on the other hand is the official language of two southern states in india, namely andhra pradesh and telangana. it is also spoken among the telugu speaking immigrant population in the usa, canada and the uk. both languages belong to the dravidian family of languages which comprise of tamil, telugu, kannada and malayalam as the major languages spoken in south india. despite belonging to the same family of languages, there are many differences between tamil and telugu, such as the script used for writing and linguistic differences in terms of phonology, morphology, syntax among others. tamil belongs to the southern branch of dravidian languages, which has a rich literary tradition spanning more than 2000 years. telugu, on the other hand, belongs to the south central branch of dravidian languages and has a considerable amount of different linguistic characteristics when compared to tamil as described by @xcite . as part of the similar language translation's subtask for dravidian languages, namely tamil (ta) and telugu (te), we have attempted to build neural machine translation (nmt) models using the opennmt-py toolkit foot_0 , which helps to generate quick prototypes for the nmt models with the desired configurations. the first nmt system (submitted as the primary system) in this work is a transformer based 6 layer encoder-decoder model which provides a single model for bidirectional translation between tamil and telugu using the datasets provided for this shared task. the second nmt system (submitted as the contrastive system) consists of two unidirectional translation models with the same configuration as the first system, but with the addition of utilizing byte pair encoding (bpe) for subword tokenization using the pre-trained multibpemb model @xcite . the rest of the work is described in sections that pertain to the related work, data, system description, results and conclusion. ## rationale for selecting the models and related work there has been a significant amount of work done on developing machine translation systems for indian languages, with some notable examples for dravidian languages such as tamil and malayalam described in @xcite . this shared task provides a unique challenge in terms of the constraint on the parallel aligned language pair data made available for training. the other challenges include the linguistically rich and domain specific content present in the prime minister of india (pmi) and the mann ki baat (mkb) datasets, where topics related to india's domestic and foreign policy issues can be found. in order to address the challenge of lengthy input (samples containing more than 300 space delimited tokens), the transformer model described by @xcite was adopted. this model provides the multi head attention mechanism which helps retain context for longer length sentence samples. to reduce the vocabulary, reduce the training time and possibly improve the translation quality (through sub word tokenization), a multibpemb model trained with a vocabulary of 100000 tokens from 275 languages has been utilised @xcite . other methods to improve translation quality, that have not been explored as part of this work are the use of back translation using monolingual corpus or corpora, on the lines of the one described by @xcite . factored nmt (which uses data tagged on the basis of morphology and parts of speech (pos)) such as the one described by @xcite is another possible candidate suitable for the kind of challenge provided by the similar language translation task, as the use of pos and morphological information can reduce the number of tokens and make the models more generalizable in terms of predictions. ## data the datasets used in the nmt systems for this work are the parallel aligned tamil and telugu (ta-te) language pairs provided as part of the dravidian language sub task of the similar language translation shared task foot_1 . some statistics about the dataset are outlined in ## system description as mentioned in section 1, the pytorch based toolkit opennmt-py has been used to create rapid prototypes for nmt models (the motivations for the same can be seen in section 2), which have then been trained on the datasets provided, validated against the provided dev sets and finally translations for the test sets described in section 3 have been obtained and submitted to the committee for evaluating the similar language translation task. a dgx station with 4 -v100 gpus have been used to train the models utilized in this task. a transformer based 6 layer encoder-decoder model on the lines of the nmt system described by @xcite , was trained for 100000 training steps as the first nmt system to be evaluated. the configuration for this model is the same as that provided by opennmt-py. in order to save time, a single bidirectional translation model for ta-te language pair has been created, which can translate from tamil to telugu and vice versa. the datasets used in this system were doubled in terms of the number of samples when compared to the second nmt system (constrastive submission), by reversing the position of the ta-te language pair and appending them to the original datasets. no special tagging identifiers were used as the tamil and telugu scripts are distinct. basic space delimited tokenization was applied on the datasets, which resulted in a combined ta-te vocabulary of 194860 tokens being generated, the relevant key configuration for this model are listed in the corpus weights help assign varied importance to the particular datasets used in this task, the values for these weights were determined after visual analysis of the dev(validation) dataset which indicated the dev dataset's contents had a greater overlap with pmi, news and (mann ki baat -which roughly translates to "from the heart") mkb in that particular order. the training time for the entire model was 18 hours. the second nmt system consists of two unidirectional translation models with the same configuration as the first system, with the addition of utilizing byte pair encoding (bpe) for subwords using the pretrained multibpemb model @xcite . the intuition behind using bpe was to reduce the vocabulary size using subword tokenization. during training it was found that the translations for the dev set couldn't distinguish between tamil and telugu subwords correctly, due to the failure in vocabulary matching for the candidates used in the evaluation and possibly due to the vocabulary shared between the languages. hence, this system was trained twice generating two unidirectional models for ta-te and te-ta translations. the training time for each model was 5 hours, which is less when compared to the primary system due to the number of samples used (the primary system uses double the number of samples) and the vocabulary size (the contrastive system has a smaller and fixed vocabulary as a pre trained bpe model has been used). ## results the evaluation metrics used to evaluate the systems in this task are bilingual evaluation understudy (bleu) score as described by @xcite , rank-based intuitive bilingual evaluation (ribes) score as described by @xcite and translation error rate (ter) as described by @xcite . corpus level metrics for the dev dataset were computed using the vizseq python library which is an implementation of several metrics described by @xcite .the metrics for the dev dataset are listed in based on the evaluation metrics of the dev (validation) dataset translations for both the systems evaluated in this work, the first system i.e. the vanilla transformer model has been submitted as the primary system. since there were no improvements in the metrics (the reason for it can be seen in section 6), during training of the second system which consists of the transformer model along with the use of multibpemb model for sub word tokenization, hence the second system has been submitted as a contrastive system. ## conclusion and future work the analysis of the evaluation metrics, from section 5, on the dev dataset indicates that the primary system, which is a transformer based encoder-decoder model, performs better than the contrastive system which contains transformer based nmt models with bpe for subword tokenization. the reason for this is possibly due to the lack of vocabulary matching the candidates being evaluated and also due to the shared vocabulary of the multibpemb model. the choice of a pre trained multibpe model was to reduce effort on the embeddings, but in hindsight training the multibpe model using the given datasets or fine tuning the pre trained multibpe model on the given datasets would have been a better choice. as seen from the evaluation of translations obtained using the dev and test datasets using bleu, ribes and ter metrics in section 5, there is a considerable scope of improvement in the scenario where a constraint is placed on the number of datasets containing parallel corpus language pair samples, that can be used for training. the possible reason for the low bleu scores in the primary system is the relatively small number of samples used along with the presence of a large variety in the linguistic forms present in the datasets. in the case of the contrastive system, the low bleu scores can be attributed to the use of the pre trained multibpe model (a pre trained bpe model fine tuned on the given datasets would have helped improved the scores). some approaches that have the potential to improve the results are, the use of back translation using monolingual corpus (through training corpus augmentation and providing more training examples for the model to learn), utilizing domain specific corpora from the shared machine translation task for indian languages described in section 2. factored nmt, an nmt which uses input tagged on the basis of morphology and parts of speech (pos) to reduce the number of tokens, the use of alternative bpe models trained on content which are a close match to the dataset used in the shared task, are other promising alternatives.