Low-Resource Machine Translation based on Asynchronous Dynamic Programming (2021)

Abstract: Reinforcement learning has been proved to be effective in handling low resource machine trans-lation tasks and different sampling methods of reinforcement learning affect the performance ofthe model. The reward for generating translation is determined by the scalability and iteration ofthe sampling strategy so it is difficult for the model to achieve bias-variance trade-off. Therefore according to the poor ability of the model to analyze the structure of the sequence in low-resourcetasks this paper proposes a neural machine translation model parameter optimization method for asynchronous dynamic programming training strategies. In view of the experience priority situa-tion under the current strategy each selective sampling experience not only improves the value ofthe experience state but also avoids the high computational resource consumption inherent in tra-ditional valuation methods (such as dynamic programming). We verify the Mongolian-Chineseand Uyghur-Chinese tasks on CCMT2019. The result shows that our method has improved the quality of low-resource neural machine translation model compared with general reinforcement learning methods which fully demonstrates the effectiveness of our method.

## introduction traditional machine translation @xcite uses cross entropy to measure the entropy of the generated token and reference token during the translation process. its training process is usually to maximize the logarithmic likelihood of each token in the target sentence by taking the source sentence and the translated target token as input. this training method is called maximum likelihood estimation @xcite . although it is easy to infer, the objective function at the token level in the training is not consistent with the evaluation indicators at the sequence level such as bleu @xcite and the model uses a given real sample word as input to predict the next word. however, since there is no real distribution in the inference stage, the model can only generate the whole sequence by predicting one word at a time based on its own prediction output. the inconsistent methods of the prediction and inference stages are easy to cause the accumulation of errors. in order to solve the problem of inconsistency and accumulation of errors, a minimum risk training (mrt) @xcite algorithm is proposed to alleviate the problem. this training algorithm introduced evaluation index as the loss function, which aims to minimize the expected loss of training data and combines sentence-level bleu into the loss function, so that the maximum likelihood estimation result of the machine translation model can be significantly improved. however, this function is not necessarily differentiable, which hinders the model from applying the bleu score directly to the training process. aiming at the above problems, reinforcement learning (rl) @xcite ) method has been applied to target optimization at the sequence level. this method directly optimises the bleu value through reinforcement training, which is a successful attempt of rl in strategy optimization at the sequence level. br-csgan @xcite is the first application of adversarial learning training to machine translation. the idea of reinforcement learning is cited in the generative adversarial network, which enables the neural machine translation model to obtain the bleu value corresponding to each possible sequence by the bootstrap method, and use this as the basis for optimizing the model parameters. in order to stabilize the training, the loss gradient of the reward is obtained by the reinforcement learning strategy through backpropagation, and finally the translation effect is steadily improved, it uses monte carlo for sampling and the sampling efficiency is improved to a certain extent.but due to the independent scalability of the monte carlo method, the sampled words are relatively independent of each other and the estimated state value does not depend on the previous state value, thereby reducing the learning accuracy. in a study @xcite , it is the first time that reinforcement learning methods were applied to machine translation, and reward shaping is used to improve the effectiveness of monolingual data in translation quality. nevertheless, the training strategy using reinforcement learning is still restricted by the setting of rewards and the efficiency of sample sampling. as mentioned above, the traditional monte carlo method has independent malleability. compared with the monte carlo method, the state value of each iteration of the agent training is determined by the previous state value and it is difficult to achieve the bias-variance trade-off. in order to achieve the trade-off, we hope that the model has the characteristics of low bias and low variance. however, the most serious challenge in the field of deep learning is the problem of high variance. the general solutions to solve the high variance problems are as follows: reduce the complexity of the model, reduce dimensionality of the data and denoise, increase the number of samples, use validation sets and regularization methods. because of the scarcity of low-resource corpus data, our method involves the use of validation sets and regularization methods to reduce variance. that is, use the validation set to observe the generalization of the training parameters and update the new state through the latest k-step state difference. the dynamic programming method further solves the problem of correlation between words, but each updated action reward needs to refer to the previous round of experience, so all the learned experience will be stored. based on the above analysis, we combined with the dynamic programming to propose an asynchronous training strategy suitable for the training of low-resource neural machine translation model. the bias-variance trade-off is achieved by selectively sampling the experience each time through the experience priority. in this paper, we used a reinforcement learning strategy combined with asynchronous dynamic programming (adp) to train machine translation models. the priortised sweeping (ps) algorithm is used to determine the experience priority. the higher the priority, the higher the priority. ultimately, we maximize the priority to generate rewards to update the strategy and use the adp method for sampling. compared with the monte carlo method, we can pay more attention to the estimation q of non-terminal state. compared with the general dynamic programming method, the k-step update method avoid the calculation disaster and the real-time supervision of model training can also avoid the accumulation of errors. it improves the accuracy rate and effectively alleviates the sparse reward problem and has a higher universal computing capability. we map the reinforcement learning algorithm to machine translation according to the markov characteristics of reinforcement learning. similar to the traditional machine translation model, we input the source language sentence into the translation model to generate the target sentence. here we use the lstm @xcite and transformer @xcite frameworks of the machine translation model based on soft attention. the second chapter of this article is about the introduction of background knowledge. the third chapter is divided into framework, asynchronous strategy and training. the fourth chapter is the experiment and analysis part, and the fifth chapter is the conclusion. ## approach in this section, we describe in detail the asynchronous strategy of nmt model training, apply the reinforcement learning algorithm to the neural machine translation model, use the adp strategy to train the gradient and update the network with parameterization. ## conclusion since different reinforcement learning sampling methods affect the performance of the model and in order to alleviate the quality problem of generated translations in low-resource machine translation tasks, this paper proposes an asynchronous dynamic sampling strategy based on reinforcement learning. this strategy selectively samples and stores each time according to the experience priority situation, thereby reducing the consumption of computing resources and improving the value of the experience state. experimental results show that this method can effectively improve the performance of low-resource machine translation. in the next step, we will combine more low-resource language features to explore more suitable for low-resource machine translation, so as to further improve the performance of low-resource machine translation systems.