Multi-Reward based Reinforcement Learning for Neural Machine Translation (2020)

Abstract: Reinforcement learning (RL) has made remarkable progress in neural machine translation (NMT). However, it exists the problems with uneven sampling distribution, sparse rewards and high variance in training phase. Therefore, we propose a multi-reward reinforcement learning training strategy to decouple action selection and value estimation. Meanwhile, our method combines with language model rewards to jointly optimize model parameters. In addition, we add Gumbel noise in sampling to obtain more effective semantic information. To verify the robustness of our method, we not only conducted experiments on large corpora, but also performed on low-resource languages. Experimental results show that our work is superior to the baselines in WMT14 English-German, LDC2014 Chinese-English and CWMT2018 Mongolian-Chinese tasks, which fully certificates the effectiveness of our method.

## introduction neural machine translation (nmt) @xcite @xcite has drawn universal attention without the demand of numerous manual work. in training phase, generic nmt models employ maximum likelihood estimation (mle) @xcite , which is the token-level objective function. however, it is inconsistent with sequence-level evaluation metrics such as bleu @xcite . reinforcement learning (rl) are leveraged for sequence generation tasks including nmt to optimize sequence-level objectives, such as actor-critic @xcite and minimum risk training (mrt) @xcite . in machine translation community, wu el at. @xcite provide the first comprehensive study of different aspects of rl training, they set a single reward to mitigate the inconsistency, and combine mle with rl to stabilize the training process. nevertheless, nmt based on reinforcement learning (rl) is unable to guarantee that the machine-translated sentences are as natural, sufficient and accurate as reference. to obtain smoother translation results, generative adversarial network (gan) and deep reinforcement learning (drl) @xcite are employed to nmt. and @xcite utilizes sentence-level bleu q as a reinforcement target based on the work of @xcite to enhance the capability of the generator. although this nova machine translation learning paradigm based on gan and drl reveals excellent manifestation, there are still some limitations: (1) when calculating rewards, the overestimation of q value will give rise to a suboptimal strategy update. (2) during training phase, it exists the problems with uneven sampling distribution, sparse rewards and high variance. what's more, the generator uses monte carlo to simulate the entire sentence, but it usually requires more calculation steps, resulting in too many parameters. (3) traditional nmt usually utilizes deterministic algorithms such as beam search or greedy decoding when predicting the next token. these methods lacks randomness, which may cause the potential best solution to be discarded. in this paper, we propose some measures to address the above problems. foremost, we adopt a novel multi-reward reinforcement learning method. that is, we weighted sum the actual reward of the discriminator, the language model reward and the sentence bleu to obtain the total reward. among them, we adopt reward shaping to alleviate the sparse reward when calculating sentence rewards. nextly, our method employ temporal-difference learning (td) @xcite to simulate the entire sentence. it effectively speeds up training and relieves the problem of error accumulation. finally, we adopt gumbel-top-k stochastic beam search @xcite to predict the next token. the method trains model more efficiently by adding the noise obeying gumbel distribution to control random sampled noise. experiments on the datasets of the english-german, chinese-english and mongolian-chinese translation tasks reveal our approach outperforms the best published results. in summary, we mainly made the following contributions: ## background & related work common nmt models are based on an encoder decoder architecture. the encoder reads and encodes the source language sequence @xmath0, ..., x n ) into the context vector representation, and the decoder generates the corresponding target language sequence ŷ = (ŷ 1 , ..., ŷm ). given h training sentence pairs x i , y i h @xmath1, at each timestep t, nmt is trained by maximum likelihood estimation(mle) and generates the target words ŷt by maximum the probability of translation conditioned on the source sentence x. the training goal is to maximize: ## approach in this section, we describe the multi-reward of reinforcement learning evaluation paradigm based on gan model. the overall architecture is shown in figure 1 . we introduce the generator g, discriminator d, sampling with gumbel-top-k stochastic beam search, calculating final reward and training the entire model in detail. ## experiment and analysis we evaluate chinese-english (zh-en), english-german (en-de) and mongolian-chinese(mo-zh) tasks to verify the effectiveness of our mr-gan. ## conclusion in this paper, we propose a novel multi-reward reinforcement learning training paradigm to guide the optimization of model parameters, which makes the reward calculation more extensive. in addition, we employ gumbel method instead of traditional beam search to selectively sample more random datas in the target space, and combining td to calculate real-time reward. we validate the effectiveness of our method on the rnnsearch and the transformer. a large number of experiments clearly show that our approach achieves significant improvements.