A Preordered RNN Layer Boosts Neural Machine Translation in Low Resource Settings (2022)

Abstract: Neural Machine Translation (NMT) models are strong enough to convey semantic and syntactic information from the source language to the target language. However, these models are suffering from the need for a large amount of data to learn the parameters. As a result, for languages with scarce data, these models are at risk of underperforming. We propose to augment attention based neural network with reordering information to alleviate the lack of data. This augmentation improves the translation quality for both English to Persian and Persian to English by up to 6% BLEU absolute over the baseline models.

## introduction nmt has recently shown promising results in machine translation @xcite @xcite . in statistical machine translation (smt), the problem is decomposed into sub-models and each individual model is trained separately, while nmt is capable of training an end-to-end model. for instance, in smt the reordering model is a feature that is trained separately and is used jointly with other features to improve the translation, while in nmt it is assumed that the model will learn the order of the words and phrases itself. sequence-to-sequence nmt models consist of two parts, an encoder to encode the input sequence to the hidden state and a decoder that decodes the hidden state to get the output sequence @xcite . the encoder model is a bidirectional recurrent neural network (rnn), the source sentence is processed once from the beginning to the end and once in parallel from the end to the beginning. one of the ideas that have not been well-explored in nmt so far is the use of existing reordering models in smt. we propose to add another layer to the encoder that includes reordering information. the intuition behind our proposal comes from the improvement achieved by bidirectional encoder model. if processing the source sentence in both directions help sequence-to-sequence model to learn better representation of the context in hidden states, adding the order of the input words as they are appearing in the output sequence as another layer may also help the model to learn a better representation in both context vectors and hidden states. in this paper we investigate this hypothesis that another layer in the encoder to process a preordered sentence can outperform both encoder architecture with two or three rnn layers. we empirically show in the experiments that adding the reordering information to nmt can improve the translation quality when we are in shortage of data. there are a few attempts to improve the smt using neural reordering models @xcite @xcite . in @xcite , three distortion models have been studied to incorporate the word reordering knowledge into nmt. they used reordering information to mainly improve the attention mechanism. in this paper, we are using a soft reordering model to improve the bidirectional attention based nmt. this model consists of two different parts. the first part is creating the soft reordering information using the input and output sequence, the second part is using this information in the attention based nmt. the rest of the paper is as follow, in section 2 a review of sequence-to-sequence nmt is provided, in section 3 the preordered model is proposed, section 4 explains the experiments and results, and section 5 concludes the paper. the notation c i is the context vector for output word y i . the context vector is the weighted sum of the hidden states as follow: the weights in this model are normalized outputs of the alignment model which is a feed-forward neural network. it uses s i-1 and h j as input and outputs a score e ij . this score is then normalized and used as the weight for computing the context vector as follow: in the encoder, a bidirectional neural network is used to produce the hidden state h. for each input word x i there is a forward and a backward hidden state computed as follow respectively: forward and backward hidden states are then concatenated to produce the hidden state h i as follow: the attention-based model is able to address some of the shortcomings of the simple encoder-decoder model for machine translation. it works fine when we have plenty of data. but if we are in lack of data the attention-based model suffers from lack of information for tuning all the parameters. we can use some other information of the input data to inject into the model and get even better results. in this paper, a model is proposed using reordering information of the data set to address the issue of shortage of data. adding this information to the model, it can improve the attention-based nmt significantly. ## conclusion in this paper we analyzed adding reordering information to nmts. nmts are strong because they can translate the source language into target without breaking the problem into sub problems. in this paper we proposed a model using explicit information which covers the hidden feature like reordering. the improvements is the result of adding extra information to the model, and helping the neural network learn the parameters in case of scarce data better.