  Reinforcement Learning (RL) is a powerful framework to address the
discrepancy between loss functions used during training and the final
evaluation metrics to be used at test time. When applied to neural Machine
Translation (MT), it minimises the mismatch between the cross-entropy loss and
non-differentiable evaluation metrics like BLEU. However, the suitability of
these metrics as reward function at training time is questionable: they tend to
be sparse and biased towards the specific words used in the reference texts. We
propose to address this problem by making models less reliant on such metrics
in two ways: (a) with an entropy-regularised RL method that does not only
maximise a reward function but also explore the action space to avoid peaky
distributions; (b) with a novel RL method that explores a dynamic unsupervised
reward function to balance between exploration and exploitation. We base our
proposals on the Soft Actor-Critic (SAC) framework, adapting the off-policy
maximum entropy model for language generation applications such as MT. We
demonstrate that SAC with BLEU reward tends to overfit less to the training
data and performs better on out-of-domain data. We also show that our dynamic
unsupervised reward can lead to better translation of ambiguous words.
