Aligning Neural Machine Translation Models: Human Feedback in Training and Inference (2024)

Abstract: Reinforcement learning from human feedback (RLHF) is a recent technique to improve the quality of the text generated by a language model, making it closer to what humans would generate.A core ingredient in RLHF’s success in aligning and improving large language models (LLMs) is its \textit{reward model} , trained using human feedback on model outputs. In machine translation (MT), where metrics trained from human annotations can readily be used as reward models, recent methods using \textit{minimum Bayes risk} decoding and reranking have succeeded in improving the final quality of translation.In this study, we comprehensively explore and compare techniques for integrating quality metrics as reward models into the MT pipeline. This includes using the reward model for data filtering, during the training phase through RL, and at inference time by employing reranking techniques, and we assess the effects of combining these in a unified approach.Our experimental results, conducted across multiple translation tasks, underscore the crucial role of effective data filtering, based on estimated quality, in harnessing the full potential of RL in enhancing MT quality.Furthermore, our findings demonstrate the effectiveness of combining RL training with reranking techniques, showcasing substantial improvements in translation quality.

## introduction neural machine translation (nmt) models @xcite are typically trained with maximum likelihood estimation (mle), maximizing the log-probability of the next word in a translation given the previous words and the source sentence. while this approach has been effective at training high-quality mt systems, the difference between the training and inference objective can lead to exposure bias @xcite @xcite , which hinders the model's ability to recover from early mistakes. furthermore, the suitability of model likelihood as a proxy for generation quality has been questioned in machine translation @xcite and beyond @xcite . these challenges sparked interest in alternative training and decoding paradigms for mt, such as reinforcement learning (rl; @xcite ) or minimum bayes risk decoding (mbr; @xcite ). more recently, the widespread success of reinforcement learning from human feedback @xcite has highlighted the importance of a good reward model that approximates well to human preferences for the task at hand. while, in general, this requires training a reward model from scratch for the specific problem, in the case of machine translation (mt), the evaluation community has achieved significant progress in developing automatic quality estimation and evaluation metrics learned from human quality annotations (e.g. comet-qe @xcite , comet @xcite , bleurt @xcite , which can be repurposed as reward models. as a consequence, recent research integrating these metrics into the training @xcite or decoding @xcite procedures has had considerable success in improving the quality of translations. however, none of the previous work has systematically compared the effect of integrating metrics at different stages of the mt pipeline or has attempted to combine these techniques in a unified approach. in this work, we perform a comprehensive study on the integration of mt quality metrics into the mt pipeline as reward models. as illustrated in figure 1 , we assess their use at different stages: as a means for data filtering, during the training process through rl, and at inference time by way of reranking techniques. furthermore, we explore the results of combining these methods. we attempt to answer the following research questions: our main contributions arise from the research questions mentioned above: ## conclusion our thorough analysis of feedback integration methods underscores the importance of meticulous data curation for enhancing mt reliability and efficiency. our findings demonstrate the consistent improvement in translation quality when employing neural metrics, such as comet(-qe), during training and/or inference. rl training with data filtering stands out as significantly superior to both mle and reranking methods. additionally, coupling rl training with reranking techniques can further enhance translation quality. while computational efficiency remains a concern due to the added overhead of rl and reranking methods on top of mletrained models, their adoption should be tailored to specific task and environmental requirements.