Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model (2024)

Abstract: Insufficient modeling of human preferences within the reward model is a major obstacle for leveraging human feedback to improve translation quality. Fortunately, quality estimation (QE), which predicts the quality of a given translation without reference, has achieved impressive alignment with human evaluations in the last two years. In this work, we investigate the potential of employing the QE model as the reward model to predict human preferences for feedback training. We first identify the overoptimization problem during QE-based feedback training, manifested as an increase in reward while translation quality declines. We examine the problem and argue that the vulnerability of the QE model might lead to high rewards for incorrect translations, resulting in overoptimization and error propagation. To address the problem, we adopt a simple yet effective method that uses heuristic rules to detect the incorrect translations and assigns a penalty term to the reward scores of them. Experimental results show that the proposed QE-based feedback training achieves consistent and significant improvements across various settings, further verified through human preference studies. Our subsequent analysis demonstrates the high data efficiency of the proposed QE-based feedback training: it outperforms systems using larger parallel corpora by a small amount of monolingual data. Our code is available at: https://github.com/zwhe99/FeedbackMT

## introduction human feedback has greatly contributed to recent advances in large language models (llms), aligning model behavior with human preferences and thereby enhancing the helpfulness and harmlessness of llms @xcite @xcite . the common practice involves using human evaluation data to train a reward model as a proxy for human preferences, followed by feedback training to fine-tune the llm and maximize the reward score. early efforts in neural machine translation (nmt) also attempted to integrate feedback to improve translation quality. most works used similarity scores (such as sentence-level bleu; @xcite between the predicted translation and a reference translation to simulate feedback rather than employing feedback from humans @xcite @xcite @xcite @xcite . few attempts to use real human feedback have been made, and these efforts either used implicit feedback in limited scenarios (e.g., e-commerce; @xcite or relied only on a minimal amount of human feedback data @xcite . therefore, the integration of real human feedback in nmt has been constrained by inadequate modeling of human preferences. fortunately, the field of mt has seen substantial advancements in quality estimation (qe; @xcite @xcite . a qe model offers a reference-free estimation of translation quality and has been facilitated by the growing availability of human evaluation data @xcite @xcite and the development of pre-trained language models @xcite . typically, given a source sentence and its translation, a sentence-level qe model can provide a numerical score to indicate the quality of the translation. the most advanced qe models to date have achieved impressive alignment with human evaluations @xcite and have been used to guide the decoding process @xcite . in light of this progress, we explore the potential of utilizing qe models as proxies of human preferences and functioning them as reward models in feedback training for the first time. firstly, we identify the overoptimization problem in feedback training, manifested as an increase in reward while translation quality declines. our analysis reveals that the underlying issue lies in the vulnerability of qe-based reward models, which, in rare instances, assign high scores to patently incorrect translations. as a result, these flawed patterns spread through subsequent training, leading to divergence from human preferences and a training collapse. however, the reward keeps rising throughout the whole process. this phenomenon aligns with the observation from @xcite that "overoptimizing" against an imperfect reward model can lead to systems "that receive good feedback from the model, but not humans". through manual examination of these flawed patterns, we categorize the most prevalent errors into two groups: length ratio errors and off-target errors (i.e., not the desired target language). guided by this observation, we propose a simple yet effective solution to mitigate overoptimization. we first detect these errors and then add a negative penalty to the reward for these erroneous translations. we show that this approach significantly alleviates the overoptimization problem and results in notable improvements across various settings, which are further verified by human preference studies 1 in summary, the contributions of this work are detailed as follows: ## conclusion we explore the potential of using the current qe model as a reward model. by identifying, analyzing, and mitigating the overoptimization problem, we successfully integrate the qe model into feedback training to refine translation quality. we validate its effectiveness across various settings, including human evaluation. further analysis demonstrates the high data efficiency of feedback training using the qe-based reward model. lastly, we delve into the impact of the base model and hyperparameters on feedback training.