Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021)

Abstract: Policy gradient algorithms have found wide adoption in NLP, but have recently become subject to criticism, doubting their suitability for NMT. Choshen et al. (2020) identify multiple weaknesses and suspect that their success is determined by the shape of output distributions rather than the reward. In this paper, we revisit these claims and study them under a wider range of configurations. Our experiments on in-domain and cross-domain adaptation reveal the importance of exploration and reward scaling, and provide empirical counter-evidence to these claims.

## introduction in neural sequence-to-sequence learning, in particular neural machine translation (nmt), reinforcement learning (rl) has gained attraction due to the suitability of policy gradient (pg) methods for the end-to-end training paradigm @xcite @xcite @xcite . the idea is to let the model explore the output space beyond the reference output that is used for standard cross-entropy minimization, by reinforcing model outputs according to their quality, effectively increasing the likelihood of higher-quality samples. the classic exploration-exploitation dilemma from rl is addressed by sampling from a pretrained model's softmax distribution over output tokens, such that the model entropy steers exploration. for the application of nmt, it was firstly utilized to bridge the mismatch between the optimization for token-level likelihoods during training and the corpus-level held-out set evaluations with nondifferentiable/decomposable metrics like bleu @xcite , and secondly to reduce exposure bias in autoregressive sequence generators @xcite . it has furthermore been identified as a promising tool to adapt pretrained models to new domains or user preferences by replacing reward functions with human feedback in humanin-the-loop learning @xcite . recently, the effectiveness of these methods has been questioned: @xcite identify multiple theoretical and empirical weaknesses, leading to the suspicion that performance gains with rl in nmt are not due to the reward signal. the most surprising result is that the replacement of a meaningful reward function (giving higher rewards to higher-quality translations) by a constant reward (reinforcing all model samples equally) yields similar improvements in bleu. to explain this counter-intuitive result, @xcite conclude that a phenomenon called the peakiness effect must be responsible for performance gains instead of the reward. this means that the most likely tokens in the beginning gain probability mass regardless of the rewards they receive during rl training. if this hypothesis was true, then the perspectives for using methods of rl for encoding real-world preferences into the model would be quite dire, as models would essentially be stuck with whatever they learned during supervised pretraining and not reflect the feedback they obtain later on. however, the analysis by choshen et al. (2020) missed a few crucial aspects of rl that have led to empirical success in previous works: first, variance reduction techniques such as the average reward baseline were already proposed with the original policy gradient by @xcite , and proved effective for nmt @xcite . second, the exploration-exploitation trade-off can be controlled by modifying the sampling function @xcite , which in turn influences the peakiness. we therefore revisit the previous findings with nmt experiments differentiating model behavior between in-domain and out-of-domain adap-tation, controlling exploration, reducing variance, and isolating the effect of reward scaling. this allows us to establish a more holistic view of the previously identified weaknesses of rl. in fact, our experiments reveal that improvements in bleu can not solely be explained by increased peakiness, and that simple methods encouraging stronger exploration can successfully move previously lower-ranked token into higher ranks. we observe generally low empirical gains in indomain adaptation, which might explain the surprising success of constant rewards in @xcite . however, we find that rewards and their scaling do matter for domain adaptation. furthermore, our results corroborate the auspicious findings of @xcite that rl mitigates exposure bias. our paper thus reinstates the potential of rl for model adaptation in nmt, and puts previous pessimistic findings into perspective. the code for our experiments is publicly available. foot_0 ## rl for nmt the objective of rl in nmt is to maximize the expected reward for the model's outputs with respect to the parameters θ: the benefit of eq. 1 is that it does not require differentiation of ∆ which allows for direct optimization of the bleu score or human feedback. foot_1 ## experiments we implement pg and mrt (without enforcing gold tokens in s; @xmath0. we simulate rewards for training samples from iwslt14 de-en with sacrebleu @xcite , and test on iwslt14 held-out sets. we consider two different domains for pretraining, wmt15 and iwslt14. this allows us to distinguish the effects of rl in in-domain learning vs domain adaptation scenarios. rl experiments are repeated three times and we report mean and standard deviation. remaining experimental details can be found in the appendix. the goal is not to find the best model in a supervised domain adaptation setup ("fine-tuning" in ## conclusion we provided empirical counter-evidence for some of the claimed weaknesses of rl in nmt by untying bleu gains from peakiness, showcasing the upwards mobility of low-ranking tokens, and re-confirming the importance of reward functions. the affirmed gains of pg variants in adaptation scenarios and their responsiveness to reward functions, combined with exposure bias repair and avoidance of the beam curse, rekindle the potential to utilize them for adapting models to human preferences. model dev bleu test bleu pretraining 34.26 34.12 pg 34.37 ± 0.12 34.24 ± 0.04 pg + average bl 34.75 ± 0.06 34.40 ± 0.03 pg + τ = 1.2 34.46 ± 0.10 34.15 ± 0.02 pg + τ = 0.8 34.47 ± 0.12 34.26 ± 0.11 pg + constant 34.26 ± 0 34.13 ± 0.01 pg + scaled 34.55 ± 0.10 34.30 ± 0.10