Comparing BERT -based Reward Functions for Deep Reinforcement Learning in Machine Translation (2022)

Abstract: In text generation tasks such as machine translation, models are generally trained using cross-entropy loss. However, mismatches between the loss function and the evaluation metric are often problematic. It is known that this problem can be addressed by direct optimization to the evaluation metric with reinforcement learning. In machine translation, previous studies have used BLEU to calculate rewards for reinforcement learning, but BLEU is not well correlated with human evaluation. In this study, we investigate the impact on machine translation quality through reinforcement learning based on evaluation metrics that are more highly correlated with human evaluation. Experimental results show that reinforcement learning with BERT-based rewards can improve various evaluation metrics.

## introduction sequence-to-sequence models based on deep learning, such as attention-based lstm @xcite and transformer @xcite , are capable of generating fluent sentences and have been used successfully in many text generation tasks, such as machine translation @xcite ) and text simplification @xcite . most previous studies on text generation use cross-entropy loss between references and output sentences to train the models based on maximum likelihood estimation for each token. differentiability of cross-entropy loss enables gradient-based estimation in a supervised learning framework, but it has a loss-evaluation mismatch problem @xcite in case of machine translation, where loss functions and evaluation metrics are not consistent, e.g., cross-entropy loss vs. bleu @xcite . that is, an output sentence that is semantically adequate may receive an unfairly low evaluation due to a superficial disagreement with the reference sentence. ## reinforcement learning for machine translation in this study, pre-trained machine translation models are fine-tuned by deep reinforcement learning using various evaluation metrics as rewards. section 2.1 describes pre-training of the machine translation model, followed by fine-tuning with reinforcement learning in section 2.2, and finally, section 2.3 outlines a machine translation metrics as a reward function for reinforcement learning. ## conclusion in this study, we investigated bert-based evaluation metrics as rewards for reinforcement learning in machine translation. the evaluation metrics can be used for both reward calculation and performance evaluation of machine translation. in the experiments, we examined the evaluation metrics in the total combination of using it as a reward and using it as a performance evaluation. experimental results on german-to-english translation of iwslt-2014 show that reinforcement learning using bert fine-tuned on sts task as a reward (sts bert) can improve performance on many of evaluation metrics. the correlation between sts bert and other evaluation metrics was relatively low, and this indicates that the high performance of sts bert was unlikely due to the effect of metric compatibility. however, sts bert has a relatively low correlation with human evaluation in the wmt-2017 metrics task and is not a good evaluation metric from this perspective. bertscore and bleurt have high correlations with human evaluation and relatively high correlations with other evaluation metrics, and also improved all metrics as rewards for reinforcement learning. therefore these metrics can also be considered good rewards. as future work, we plan to use quality estimation @xcite without reference sentences as a reward for reinforcement learning of machine translation. rewards based on quality estimation have the potential to improve machine translation models in an unsupervised manner.