Attention Transformer Model for Translation of Similar Languages (2020)

Abstract: This paper illustrates our approach to the shared task on similar language translation in the fifth conference on machine translation (WMT-20). Our motivation comes from the latest state of the art neural machine translation in which Transformers and Recurrent Attention models are effectively used. A typical sequence-sequence architecture consists of an encoder and a decoder Recurrent Neural Network (RNN). The encoder recursively processes a source sequence and reduces it into a fixed-length vector (context), and the decoder generates a target sequence, token by token, conditioned on the same context. In contrast, the advantage of transformers is to reduce the training time by offering a higher degree of parallelism at the cost of freedom for sequential order. With the introduction of Recurrent Attention, it allows the decoder to focus effectively on order of the source sequence at different decoding steps. In our approach, we have combined the recurrence based layered encoder-decoder model with the Transformer model. Our Attention Transformer model enjoys the benefits of both Recurrent Attention and Transformer to quickly learn the most probable sequence for decoding in the target language. The architecture is especially suited for similar languages (languages coming from the same family). We have submitted our system for both Indo-Aryan Language forward (Hindi to Marathi) and reverse (Marathi to Hindi) pair. Our system trains on the parallel corpus of the training dataset provided by the organizers and achieved an average BLEU point of 3.68 with 97.64 TER score for the Hindi-Marathi, along with 9.02 BLEU point and 88.6 TER score for Marathi-Hindi testing set.

## introduction this paper focuses on establishing a neural machine translation model using encoder-decoder ar-chitecture to translate between hindi-marathi sentence pairs. we are utilizing an attention mechanism approach by employing the combination of recurrence based rnn encoder-decoder layers @xcite and latest machine translation technique the transformers @xcite to address the complexity of wmt-20 similar language data-set (hindi-marathi). the motivation behind combining rnn and transformer architecture is taken from the paper @xcite to utilize the benefits of both. the formulated machine translation system has numerous applications. it can be used in advertising campaigns to reach native users. the media industry can employ this technology for generating subtitles and broadcasting multilingual news to cover a wide range of native subscribers of a region. moreover, these systems can provide native vernaculars in social media to increase the activities of indigenous individuals of a native language. additionally, search engines can also adopt this method to display relevant results to clients in their region-specific native language. the paper is composed of three further sections. the second section will display our proposed approach to solve the problem of hindi-marathi translation, followed by a discussion on designed experiments for training the model and mechanism for generating the results. lastly, we will illustrate contributions and future work. ## methodology the paper proposes a novel approach called attention transformer shown in fig. 1 to translate between hindi and marathi text. in the figure, two encoder and decoder layers are illustrated. the initial n encoder-decoder layers contain rnn units in the form of lstms, which get stacked on top of each other. first, the initial encoder layer processes the tokenized input and generates a context vec-tor. the initial decoder layer consumes the context vector generated by the neighboring encoder layer. plus, it takes the target output as its input while training by utilizing the concept of teacher force training @xcite . next, the processed outputs of both the initial encoder-decoder layer gets presented as input to the transformer model. in the transformer model, we can observe the second encoder and decoder layer stacked on the top of positional input-output embedding layers. in the transformer, the positional input-output embedding layer receives the output of the initial encoder-decoder layer as its input. the role of the positional embedding layer is first to convert its input, which comes from the previous rnn based encoder-decoder layer into d-dimensional space where d is the output size of the embedding layer and add a positional encoding vector to it. as a result, all similar words relative to their positions in the training sentences will get cluster together. the working of the positional encoding vector is presented in the paper @xcite . next, the outputs of the positional embedding layer get passed to encoder-decoder layers of transformer. an individual encoder-decoder layer of transformer contains a multi-head attention mechanism followed by a feed-forward layer, and a transformer can have n number of such layers. the multi-head attention mechanism and feed-forward layer works, as illustrated in the paper @xcite , and their outputs are normalized. to train the model, we can apply the teacher forcing mechanism while during testing, a start token is initially required for the decoder to start the decoding process. after that, we can let the decoder to generate the output tokens in a loop by utilizing its current output as input to the next time frame until it produces the end token. ## experimental studies we have used two human-level translation evaluation criteria, which are bleu @xcite and ter @xcite scores and two general evaluation metrics that are sparse categorical accuracy and mean loss. this section will first discuss the preparation of training dataset and baseline models, followed by training procedures plus their outcomes. and then, we will move towards explaining the results of the testing procedure. it is important to note that all experiments given below are performed using tpu with 180 tflops, and 64 gb high bandwidth memory (hbm) provided by google colab, plus an implementation of the experiment is located in the colab notebook (implementation). ## conclusion this paper has presented a supervised deep neural translation-based approach called attention transformer as a tool to perform translation between similar pair of languages (hindi-marathi). we have developed a novel neural translation method called attention transformer to transmute from hindi source to marathi and vice-versa by combining the classical recurrence based encoder-decoder approach and transformers working mechanisms. all supervised translation approaches need parallel corpora as their data-set to learn the probability function of generating translation from source to target. we have solely utilized the wmt-20 hindi-marathi parallel corpus as the training dataset for the attention transformer model having 44,685 sentence pairs and used two human-level evaluation criteria, bleu plus ter scores, to evaluate the attention transformer model. we have achieved bleu points of 3.68 and a ter score of 97.64 for the hindi-marathi test pair. and, bleu points of 9.02 with the ter score of 88.68 for the marathi-hindi test data-set. the future work under this domain includes applying stochastic optimizations like a genetic algorithm to find the best possible combinations of hyper-parameter to model the probability distribution of source to the target language. furthermore, we can also stack a reinforcement learning paradigm on a developed supervised neural translation model to create a selfautonomous personalized environment for learning the probability function, which continuously gets updated by taking real-time feedback from the user.