An Effective Optimization Method for Neural Machine Translation: The Case of E nglish- P ersian Bilingually Low-Resource Scenario (2020)

Abstract: In this paper, we propose a useful optimization method for low-resource Neural Machine Translation (NMT) by investigating the effectiveness of multiple neural network optimization algorithms. Our results confirm that applying the proposed optimization method on English-Persian translation can exceed translation quality compared to the English-Persian Statistical Machine Translation (SMT) paradigm.

## introduction employing neural networks in machine translation (mt) significantly reduces the time-consuming and laborious operation steps such as word alignment, phrase extraction, feature selection, etc. although the quality of neural mt (nmt) models heavily rely on the quantity as well as the quality of the training dataset, considering the low-resource condition, the impact of nmt is still not as much as the statistical machine translation (smt). nmt has recently achieved great success, which surpasses smt in many high-resource language pairs, and it has become the mt approach. in this paper, we compare the impact of multiple neural network optimization algorithms under with respect to the low-resource condition, and then, we proposes an effective optimization method for our case-study language pair. the motivation for choosing english and persian as the case-study is the linguistic differences between these languages, which are from different language families and have significant differences in their properties, may pose a challenge for mt. following @xcite , lowresource languages are those that have fewer technologies and datasets relative to some measure of their international importance. in simple words, the languages for which bilingual training data is extremely sparse, requiring recourse to techniques that are complementary to standard mt approaches. the biggest issue with low-resource languages is the extreme difficulty of obtaining sufficient resources. natural language processing (nlp) methods that have been created for analysis of lowresource languages are likely to encounter similar issues to those faced by documentary and descriptive linguists whose primary endeavor is the study of minority languages @xcite . lessons learned from such studies are highly informative to nlp researchers who seek to overcome analogous challenges in the computational processing of these types of languages. our results show that the proposed optimization algorithm for english-persian nmt works well and improves translation results compared to the english-persian smt paradigm. this paper is organized as follows; section 2 describes the methodology. the experimental results and analysis are covered by section 3. section 4 investigates the previous related work. conclusions and future work are provided in section 5. ## methodology nmt originates from sequence-to-sequence learning. so, in this paper, we take the attention-based (attentional) nmt model. ## experimental results for the experiments, we utilized tep foot_1 @xcite english-persian parallel corpus that contains about 594k sentences. we allocated ≈550k sentences to training step, ≈10k sentences to validation step, and ≈30k sentences to testing step. we employed byte pair encoding (bpe) @xcite as an effective way to overcome the unknown word problem in standard nmt. in the experiments, we limited the vocabulary size to the most frequent 10k tokens and replacing the rest with a special @xmath0. we accelerate training by discarding all sentences with more than 30 elements (either bpe units or actual tokens). the vector dimension of bilingual words is 512, the size of hidden layer is 1024, the beam size is 10, the size of mini-batch is 80, and the dropout of output layer is set to 0.1. in order to reduce the problem of unlisted words, the size of persian and english dictionaries is set to 20k to cover about 95% words. in order to reduce fitting, we set epoch, as the maximum number of training rounds, to 60. bleu @xcite is our standard evaluation metric. we employed the following experimental systems: mantis+sgd+dc represents learning rate decay when the iteration exceeds 40 rounds, and the decay rate is 0.1. as seen in the results of model 4, model 5, model 6, model 8, and model 10 demonstrate that increasing learning rate can extremely improve the english-persian translation quality where when the optimization method of sgd and adam were employed. however, when the learning rate is too high, the performance of translation system will be reduced. therefore, it can be seen in the case of low-resource conditions, the actual nmt system is sensitive in various model optimization methods and corresponding learning rates. so, selecting the appropriate model optimization method and learning rate has a great influence on the final translation results. furthermore, model 7 has achieved the highest translation impact, which surpasses the smt system using moses and the nmt system using rnnsearch. it can be found that english-persian nmt still achieves better translation results by adopting higher learning rates and learning rate scheduling strategies with fewer corpus when choosing appropriate model optimization methods. figure 1 shows the convergence curves of different system models. ## related work to construct pseudo bilingual corpus, various useful methods have already been proposed; 1) back-translation @xcite , 2) dual learning @xcite , and 3) round-tripping @xcite . also, integrating additional language models to use monolingual corpus @xcite , using transfer learning to transfer the model of high-resource language pairs to low-resource ones, etc. the core idea of the mentioned approaches is to integrate more external resources so that the nmt model can sufficiently acquire translation knowledge and augment translation quality. although the above methods have practically achieved remarkable results, the disadvantage is that the application effect is limited by the quality of external (generated) sentences. in contrast the existing work, this paper compares various optimization methods of nmt models, and proposes a translation model optimization method which is useful in low-resource condition to enhance the effectiveness of english-persian nmt. our method does not employ any additional (generated) resources and has certain generality. ## conclusions and future work in this paper, an effective optimization method for bilingually low-resource nmt models was applied to english-persian translation. the investigated optimization method significantly enhances the impact of the english-persian nmt system, and surpasses the smt system and the previous similar work, which achieves the best translation results. noting worth that our optimization method not only does not depend on external resources but also it has language independence. as a future work, we want to investigate other methods of nmt to enhance effects in low-resource conditions, and the application of this method to other languages.