Exploring Supervised and Unsupervised Rewards in Machine Translation (2021)

Abstract: Reinforcement Learning (RL) is a powerful framework to address the discrepancy between loss functions used during training and the final evaluation metrics to be used at test time. When applied to neural Machine Translation (MT), it minimises the mismatch between the cross-entropy loss and non-differentiable evaluation metrics like BLEU. However, the suitability of these metrics as reward function at training time is questionable: they tend to be sparse and biased towards the specific words used in the reference texts. We propose to address this problem by making models less reliant on such metrics in two ways: (a) with an entropy-regularised RL method that does not only maximise a reward function but also explore the action space to avoid peaky distributions; (b) with a novel RL method that explores a dynamic unsupervised reward function to balance between exploration and exploitation. We base our proposals on the Soft Actor-Critic (SAC) framework, adapting the off-policy maximum entropy model for language generation applications such as MT. We demonstrate that SAC with BLEU reward tends to overfit less to the training data and performs better on out-of-domain data. We also show that our dynamic unsupervised reward can lead to better translation of ambiguous words.

## introduction autoregressive sequence-to-sequence (seq2seq) neural architectures have become the de facto approach in machine translation (mt). such models include recurrent neural networks (rnn) @xcite and transformer networks @xcite , among others. however, these models have as a serious limitation the discrepancy between their training and inference time regimes. they are traditionally trained using the maximum likelihood estimation (mle), which aims to maximise log-likelihood of a categorical ground truth distribution (samples in the training corpus) using loss functions such as cross-entropy, which are very different from the evaluation metric used at inference time, which generally compares string similarity between the system output and reference outputs. moreover, during training, the generator receives the ground truth as input and is trained to minimise the loss of a single token at a time without taking the sequential nature of language into account. at inference time, however, the generator will take the previous sampled output as the input at next time step, rather than the ground truth word. mle training thus causes: (a) the problem of "exposure bias" as a result of recursive conditioning on its own errors at test time, since the model has never been exclusively "exposed" to its own predictions during training; (b) a mismatch between the training objective and the test objective, where the latter relies on evaluation using discrete and non-differentiable measures such as bleu @xcite . the current solution for both problems is mainly based on reinforcement learning (rl), where a seq2seq model @xcite is used as the policy which generates actions (tokens) and at each step receives rewards based on a discrete metric taking into account importance of immediate and future rewards. however, rl methods for seq2seq mt models also have their challenges: high-dimensional discrete action space, efficient sampling and exploration, choice of baseline reward, among others @xcite . the typical metrics used as rewards (e.g., bleu) are often biased and sparse. they are measured against one or a few human references and do not take into account alternative translation options that are not present in the references. one way to address this problem is to use entropy-regularised rl frameworks. they incorporate the entropy measure of the policy into the reward to encourage exploration. the expectation is that this leads to learning a policy that acts as stochastically as possible while able to succeed at the task. specifically, we focus on the soft actor-critic (sac) @xcite ) rl framework, which to the best of our knowledge has not yet been explored for mt, as well as other natural language processing (nlp) tasks. the main advantage of this architecture, as compared to other entropy regularised architectures @xcite , is that it is formulated in the off-policy setting that enables reusing previously collected samples for more stability and better exploration. we demonstrate that sac prevents the model from overfitting, and as a consequence leads to better performance on out-of-domain data. another way to address the problem of sparse or biased reward is to design an unsupervised reward. recently, in robotics, sac has been successfully used in unsupervised reward architectures, such as the "diversity is all you need" (diayn) framework @xcite . diayn allows the learning of latent-conditioned sub-policies ("skills") in unsupervised manner, which allows to better explore and model target distributions. inspired by this work, we propose a formulation of an unsupervised reward for mt. we thoroughly investigate effects of this reward and conclude that it is useful in lexical choice, particularly the rare sense translation for ambiguous words. our main contributions are thus twofold: (a) the re-framing of the sac framework such that it can be applied to mt and other natural language generation tasks (section 3). we demonstrate that sac results in improved generalisation compared to the mle training, leading to better translation of out-of-domain data; (b) the proposal of a dynamic unsupervised reward within the sac framework (section 3.4). we demonstrate its efficacy in translating ambiguous words, particularly the rare senses of such words. our datasets and settings are described in section 4, and our experiments in section 5. ## related work reinforcement learning for mt rl has been successfully applied to mt to bridge the gap between training and testing by optimising the sequence-level objective directly @xcite @xcite . however, thus far mainly the reinforce (williams, 1992) algorithm and its variants have been used @xcite . these are simpler algorithms that handle the large natural language action space, but they employ a sequencelevel reward which tends to be sparse. to reduce model variance, actor-critic (ac) models consider the reward at each decoding step and use the critic model to guide future actions @xcite . this approach has also been explored for mt @xcite . however, more advanced ac models with q-learning are rarely applied to language generation problems. this is due to the difficulty of approximating the q-function for the large action space. the large action space is one of the bottleneck for rl for text generation in general. pre-training of the agent parameters to be close to the true distribution is thus necessary to make rl work @xcite . further rl training of the agent makes the overfitting problem even more pronounced resulting in peaky distributions. such problems are traditionally addressed by entropy regularised rl. entropy regularised rl the main goal of this type of rl is to learn an efficient policy while keeping the entropy of the agent actions as high as possible. the paradigm promotes exploration of actions, suppresses peaky distributions and improves robustness. in this work, we explore the effectiveness of the maximum entropy sac framework @xcite . the work closest to ours is of @xcite where the entropy-regularised ac (erac) model leads to better mt performance. the major difference between erac and sac is that the former is an on-policy model and the latter is an off-policy model. on-policy approaches use consecutive samples collected in real-time that are correlated to each other. in the off-policy setting, our sac algorithm uses samples from the memory that are taken uniformly with reduced correlation. this key characteristic of sac ensures better model generalisation and stability @xcite . there are also differences in the architectures of sac and erac, i.a., using 4 q-value networks instead of two. these differences will be covered in detail in section 3. unsupervised reward rl significant work has been done in robotics to improve the learning capability of robots. these approaches do not rely on a single objective but rather promote intrinsic motivation and exploration. such an approach to learn diverse skills (latent-conditioned sub-policies, in practice, skills like walking or jumping) in unsupervised manner was recently proposed by @xcite . the approach relies on the sac model and inspired our approach to designing our unsupervised reward for mt. we are not aware of other attempts to design dynamic unsupervised rl rewards (learnt together with the network) in seq2seq in general, or mt in particular. recent work on unsupervised rewards in nlp @xcite explores mainly static rewards computed against synthetic references. ## methodology in this section we start by describing the underlying mt architecture and its variant using rl, to then introduce our sac formulation and the reward functions used. ## conclusions we propose and reformulate sac reinforcement learning approaches to help machine translation through better exploration and less reliance on the reward function. to provide a good trade-off between exploration and quality, we devise two reward methods in the supervised and dynamic unsupervised manner. the maximum entropy off-policy sac algorithm mitigates the overfitting problem when evaluated in the out-of-domain space; both rewards introduced in our sac architecture can achieve better quality for lexical translation of ambiguous words, particularly the rare senses of en-fr source word hill gold target word colline source sentence the teen jumps the hill with his bicycle . reference sentence ado saute sur la colline 'hill' avec son vélo . mle adolescent saute sur la pente 'slope' avec son vélo . sac bleu adolescent saute la pente 'slope' avec son vélo . sac unsuper adolescent saute la colline 'hill' avec son vélo . en-de source word outfit gold target word outfit source sentence a rhythmic gymnast in a blue and pink outfit performs a ribbon routine . reference sentence eine rhythmische sportgymnastin in einem blauen und pinken outfit vollführt eine bewegung mit dem band . mle ein begeisterter turner in blau-rosa kleidung 'dress' führt eine band auf . sac bleu ein begeisterter turner in blau-rosa kleidung 'dress' führt eine band auf . sac unsuper ein aufgeregter turner in einem blau-rosa outfit führt eine band aus .