P unjabi to E nglish Bidirectional NMT System (2020)

Abstract: Machine Translation is ongoing research for last few decades. Today, Corpus-based Machine Translation systems are very popular. Statistical Machine Translation and Neural Machine Translation are based on the parallel corpus. In this research, the Punjabi to English Bidirectional Neural Machine Translation system is developed. To improve the accuracy of the Neural Machine Translation system, Word Embedding and Byte Pair Encoding is used. The claimed BLEU score is 38.30 for Punjabi to English Neural Machine Translation system and 36.96 for English to Punjabi Neural Machine Translation system.

## introduction machine translation (mt) is a popular topic in natural language processing (nlp). mt system takes the source language text as input and translates it into target-language text @xcite . various approaches have been developed for mt systems, for example, rulebased, example-based, statistical-based, neural network-based, and hybrid-based @xcite . among all these approaches, statistical-based and neural network-based approaches are most popular in the community of mt researchers. statistical and neural network-based approaches are data-driven @xcite . both need a parallel corpus for training and validation @xcite . due to this, the accuracy of these systems is higher than the rule-based system. the neural machine translation (nmt) is a trending approach these days @xcite . deep learning is a fast expanding approach to machine learning and has demonstrated excellent performance when applied to a range of tasks such as speech generation, dna prediction, nlp, image recognition, and mt, etc. in this nlp tools demonstration, punjabi to english bidirectional nmt system is showcased. the nmt system is based on the sequence to sequence architecture. the sequence to sequence architecture converts one sequence into another sequence @xcite . for example: in mt sequence to sequence, architecture converts source text (punjabi) sequence to target text (english) sequence. the nmt system uses the encoder and decoder to convert input text into a fixed-size vector and generates output from this encoded vector. this encoder-decoder framework is based on the recurrent neural network (rnn) @xcite . this basic encoder-decoder framework is suitable for short sentences only and does not work well in the case of long sentences. the use of attention mechanisms with the encoderdecoder framework is a solution for that. in the attention mechanism, attention is paid to subparts of sentences during translation. ## corpus development for this demonstration, the punjabi-english corpus is prepared by collecting from the various online resources. different processing steps have been done on the corpus to make it clean and useful for the training. the parallel corpus of 259623 sentences is used for training, development, and testing the system. this parallel corpus is divided into training (256787 sentences), development (1418 sentences), and testing (1418 sentences) sets after shuffling the whole corpus using python code. ## pre-processing of corpus pre-processing is the primary step in the development of the mt system. various steps have been performed in the pre-processing phase: tokenization of punjabi and english text, lowercasing of english text, removing of contraction in english text and cleaning of long sentences (# of tokens more than 40). ## methodology to develop the punjabi to english bidirectional nmt system, the opennmt toolkit @xcite is used. opennmt is an open-source ecosystem for neural sequence learning and nmt. two models are developed: one for translation of punjabi to english and the second for translation of english to punjabi. the punjabi vocabulary size of 75332 words and english vocabulary size of 93458 words is developed in the pre-processing step of training the nmt system. for all models, the batch size of 32 and 25 epochs for training is fixed. for the encoder, bilstm is used, and lstm is used for the decoder. the number of hidden layers is set to four in both encode and decoder. the number of units is set to 500 cells for each layer. bpe @xcite is used to reduce the vocabulary size as the nmt suffers from the fixed vocabulary size. the punjabi vocabulary size after bpe is 29500 words and english vocabulary size after bpe is 28879 words. "general" is used as an attention function. by using python and flask, a web-based interface is also developed for punjabi to english bidirectional nmt system. this interface uses the two models at the backend to translate the punjabi text to english text and to translate english text to punjabi text. the user enters input in the given text area and selects the appropriate nmt model from the dropdown and then clicks on the submit button. the input is pre-processed, and then the nmt model translates the text into the target text. ## results both proposed models are evaluated by using the bleu score @xcite . the bleu score obtained at all epochs is recorded in a table for both models.