PICT @ D ravidian L ang T ech- ACL 2022: Neural Machine Translation On D ravidian Languages (2022)

Abstract: This paper presents a summary of the findings that we obtained based on the shared task on machine translation of Dravidian languages. As a part of this shared task, we carried out neural machine translations for the following five language pairs: Kannada to Tamil, Kannada to Telugu, Kannada to Malayalam, Kannada to Sanskrit, and Kannada to Tulu. The datasets for each of the five language pairs were used to train various translation models, including Seq2Seq models such as LSTM, bidirectional LSTM, Conv Seq2Seq, and training state-of-the-art as transformers from scratch, and fine-tuning already pre-trained models. For some models involving monolingual corpora, we implemented backtranslation as well. These models’ accuracy was later tested with a part of the same dataset using BLEU score as an evaluation metric.

## introduction often, it becomes a challenge to develop a robust bilingual machine translation system, and that too with limited resources at hand @xcite . moreover, for low-resource languages, such as the dravidian family of languages, achieving high accuracy of translations remains a concern @xcite . this paper presents the development of machine translation systems for kannada to other dravidian languages such as tamil, telugu, malayalam, tulu, and sanskrit. tamil is a dravidian classical language used by the tamil people of south asia. tamil is an official language of tamil nadu, sri lanka, singapore, and the union territory of puducherry in india @xcite @xcite . significant * equal contribution † equal contribution ‡ equal contribution minority speak tamil in the four other south indian states of kerala, karnataka, andhra pradesh, and telangana, as well as the union territory of the andaman and nicobar islands. it is also spoken by the tamil diaspora, which may be found in malaysia, myanmar, south africa, the united kingdom, the united states, canada, australia, and mauritius. tamil is also the native language of sri lankan moors @xcite @xcite @xcite . tamil, one of the 22 scheduled languages in the indian constitution, was the first to be designated as a classical language of india @xcite . malayalam is tamil's closest significant cousin; the two began splitting during the 13th century ad. although several variations between tamil and malayalam indicate a pre-historic break of the western dialect, the process of separating into a different language, malayalam, did not occur until the 15th or 17th century @xcite . one of the approaches implemented consisted of training conventional machine translation models which involved sequence to sequence learning (seq2seq) @xcite . seq2seq is an encoder-decoder approach, in which the encoder reads the input sequence, one word at a time to produce a hidden vector. the decoder produces the output sequence from the vector received from the encoder. we used lstms @xcite , bidirectional lstms (bilstm) @xcite which learns bidirectional long-term dependencies between time steps of time series or sequence data, and convolutional seq2seq learning (conv2seq) @xcite which uses multiple stacked layers of cnns to learn long term dependencies with lower time complexity. the second approach involved training the transformer model @xcite from scratch using the fairseq library @xcite . we also imple-parallel kn-ml kn-ta kn-te kn-tu kn-sn official 90,974 88,813 88,503 9,470 8,300 monolingual ml ta te te te indiccorp 80,000 80,000 80,000 -- ## dataset description the bilingual dataset provided by the organizers @xcite was divided into three sub-corporas of train, dev and test. the statistics of the training data is given in to further improve the accuracy of the translations we used back-translation. the monolingual data used for back-translation was taken from in-diccorp @xcite (a large publiclyavailable corpora for indian languages created by ai4bharat from scraping through news, magazines, and books over the web). monolingual data used was 80,000 each for malayalam, tamil and telugu. we chose 80,000 sentences according to the memory limitations of our gpu. we didn't perform backtranslation on tulu and sanskrit as we couldn't find good monolingual data for those languages. from the monolingual data taken we generate pseudo-parallel data. using the official and the pseudo-parallel data we train models to provide translations from kannada to the given indic languages. ## data preparation in data preprocessing, the sentences present in the given dataset contain punctuations, synonyms, misspelled words, numbers, etc., and they have to be cleaned before we pass it to the model. for the sentences of kannada, malayalam, tamil and telugu languages, we used the preprocessing given by indicnlp library foot_0 , which contains preprocessing for various indian languages. we normalize (helpful in reducing the number of unique tokens present in the text) and then pre-tokenize (for splitting the text object into smaller tokens for better model training) (harish and rangan, 2020) the input given followed by transliterating all the indic data written in their own corresponding scripture to devanagari scripture, along with applying byte-pair encoding (bpe) @xcite . finally we pass the data to fairseq-preprocess to binarize training data and build vocabularies from the text of that particular language. for seq2seq models such as lstm and bil-stm, we took a smaller portion of the dataset, and split it into training data of corpora size 4000, and dev and test datasets of size 1000 for each language pair. for training the seq2seq models as well as for training simple transformers from scratch, we used the sacremoses tokenization foot_1 , where sacremoses is a pre-installed dependency in the fairseq toolkit. in the first system, we download the indic-indic model for multilingual neural machine translation given by indictrans foot_2 which was trained on the highest score achieved for each language pair is marked in bold. samanantar dataset @xcite . we then generate the pseudo-translations for monolingual data using the same pre-trained transformer_4x multilingual model. finally, we train the official data and the pseudo-parallel data generated using back-translation to give the translation for the given languages. the second system which we used was a convolutional neural network (cnn) trained using using the 'fconv' architecture provided by the opensource toolkit fairseq.py. other seq2seq architectures for machine translation included lstm and bilstm, wherein lstm we construct a standard encoder-decoder lstm architecture, which is provided in the open-source toolkit fairseq.py whereas for bilstm we use the same 'lstm' architecture provided, with the only change of making the original encoder parameter as bidirectional. we also trained standard transformer models from scratch, again by using the fairseq library 4 . fairseq provides a standard transformer architecture which can be further used for training custom transformer models for machine translation. ## results we obtained rank 1 for translations from kannada to malayalam, kannada to telugu and kannada to tamil. for translations from kannada to sanskrit and for kannada to tulu translations we stood 3rd and 4th respectively (we had initially sent the wrong results for kn-sn and kn-tu for the workshop task submission, hence the low scores were obtained for the same). results of test sets on the shared task is given in ## related work the domain of neural machine translation tasks has been among the interest topics for many researchers. the first machine translation model using deep neural networks was proposed by kalchbrenner and blunsom @xcite . nmt has since been widely studied across the scientific community. in encoder-decoder mechanisms, the words are converted into word embeddings in the encoder, which are then passed to the decoder which uses an attention mechanism, encoder representations, and previous words to generate the next word in the translation. the encoder and decoder can be deep neural networks such as rnn @xcite , cnn @xcite , or feed-forward neural networks @xcite . further, there were self-attention models proposed such as transformers which aided to further research in nmt. a notable research related to the efficiency of the same was presented at the proceedings of the 7th workshop on asian translation in 2020 @xcite . other related works include those presented at previous acl conferences in 2019 @xcite and 2020 @xcite . pertaining to research in machine translation in dravidian languages, xie @xcite was able to achieve bleu scores of 38.86, 36.66, and 19.84 for english-telugu, english-tamil, and english-malayalam using multilingual translation and backtranslation. @xcite worked on implementing a translation system for english to kannada by limited use of supplementary data between english and other dravidian languages. other works include cvit's submissions to wat-2019 @xcite , a transformer-based multilingual indic-english nmt system @xcite , comparison of different orthographies for machine translation of under-resourced dravidian languages @xcite , etc. ## conclusion thus, we implemented neural machine translation systems for dravidian languages. we utilized different architectures for the same, and analyzed their performance. in future, we plan to train our models with large-scale gpus. we plan to apply other tokenization methods for the language corpora as well for better training. also, we plan to train our models with expanded corpora for better results.