Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization (2025)

Abstract: Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task–data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and through human evaluation.

## introduction for many natural language generation (nlg) tasks, aligning models to human preferences has led to large performance gains @xcite . a strong motivation for this alignment step is that much of the data on which the model was originally trained -internet text -is useful for language generation in general but does not match the desired output for the task. neural machine translation (nmt) models have not involved alignment to human preferences, in part because of the assumption that supervised training data for nmt does match the desired output of the translation task. however, we show the existence of a mismatch between the nmt task and typical training data. machine translation is unusual among nlg tasks in that task-relevant supervised training data -text paired with its translation -is plentiful and publicly available. one might expect that with such a large amount of task-relevant training data, there would be no need for task-alignment. however, we identify an exhaustive list of reasons why training examples in a parallel corpus diverge from the desired output in meaningful ways (see section 2.2). machine translation is also unusual in that human preference data has been collected and published for a large number of systems, and translation quality estimation (qe) is an active research area that has benefited greatly from recent advances in large language models. we introduce a method for using quality estimation models, which themselves are trained from human preference data, in order to perform nmt task alignment. our method, direct quality optimization (dqo), is a batched online variant of direct preference optimization (dpo) @xcite that uses a qe model as a proxy for human preference. we show that dqo improves translation quality in terms of bleu, comet22, cometkiwi22, and bleurt, and leads to a reduction in translation errors in a human evaluation using the multidimensional quality metric framework (mqm) @xcite . we make three notable observations when applying dqo to a multilingual model: while we attribute much of the performance in held-out languages to transfer learning of general behaviors required by the translation task (e.g. avoiding omissions), the language-specific improvements in held-out languages cannot be explained by transfer learning. instead, these results suggest that dqo does not only increase the likelihood of the features present in its task alignment data, but also focuses the model on human preference features that it already learned during supervised training. because of this, the task of nmt cannot be reduced to producing valid translations, nor humanlike translations, but instead requires generating human-preferred translations -those judged as at least as good as all other valid translations. ## human preference learning for llms supervised data showing chat-based dialog between humans and ai assistants was, prior to the wide availability of such agents in the form of llms, understandably rare. even with the advent of high quality proprietary and open-source models, which one could sample to create synthetic data, there is a fundamental task-data mismatch: the task is not to imitate an existing ai assistant, but (ideally) to train a new state-of-the-art model. llm training instead follows a two-step process: in step one, the actual task for which the model is optimized is predicting the next token in documents taken from the web. this, when done at scale and with a variety of data sources, provides the model with extensive world knowledge and understanding of a wide array of styles and document types. this is then followed by instruction fine-tuning, a comparatively brief round of supervised learning on human-or ai-labeled examples of dialogues, which brings the model's output distribution into the general neighborhood of desired behavior. finally human preference learning, using actual human rankings aligns the model with the desired task: producing human-preferred responses to questions and dialog, while remaining helpful and harmless @xcite . direct preference optimization (dpo) is a preference learning algorithm that trains on preference pairs of the form (x, y w , y l ), with x being a model input, and y w and y l being two potential model outputs for the input x, marked as chosen (winning) or rejected (losing) by humans during data collection @xcite , using the loss function: ## direct quality optimization for nmt because of its stability and ease of use, we select dpo as the basis for our experiments with human preference learning as a form of task alignment for nmt. as a proxy for human preferences, we use the cometkiwi22 quality estimation model to score and compare multiple translations of a given source @xcite . cometkiwi22 is highly multilingual and has been shown to correlate well with human preference @xcite . to verify that our method is not dependent on the specific choice of quality estimation model we conducted a brief experiment using metricx @xcite instead of cometkiwi22 and obtained very similar results. our main experiments are run with the nvidia megatron english-many model foot_0 , a 500m parameter encoder-decoder model, which supports translating from english into 30 languages foot_1 from 14 language families, listed in language family languages (iso 639-1) baltic lt, lv germanic da, de, nl, no, sv romance es, fr, it, pt, ro slavic bg, cs, hr, pl, ru, sl, uk uralic et, fi, hu other el, hi, id, ja, ko, tr, vi, zh the model's multilingual nature allows us to apply task alignment to a subset of language pairs and observe the effects on unrelated languages, with minimal risk of exposing the model to any new information in those languages. any improvements in those languages must either apply to all languages (such as avoiding omissions or additions), or are language specific, and can only have come from previously unused latent knowledge acquired during supervised training. in our experiments, we selected chinese, german, hindi, russian and spanish as the target languages used during task alignment, termed @xmath0, es, hi, ru, es}. let t @xmath1, r be the set of languages related to at least one language in t (defined as belonging to the same language family), and r @xmath2. an overview of how many languages belong to each set is shown in ## related work the idea of task-data mismatch in nmt is not new. there has been extensive previous work focused on reducing this mismatch through data filtering, using surface-level heuristics @xcite , statistical and neural models for alignment and quality evaluation @xcite @xcite , language identification @xcite , or ensembles @xcite . while data filtering techniques do help reduce the task-data mismatch, they force a trade-off between increasing task alignment and retaining flawed, but potentially useful, training data. to counter this, curriculum learning can be used, by training first on a conservatively filtered dataset, then shifting to a cleaner subset of the data @xcite . however, no amount of data filtering can remove the effects of translationese, as it is present in all translations. @xcite and @xcite both address this by treating original and translated text as separate languages in a "multilingual" nmt model, by training either a classifier or a contrastive language model to tag each source and target segment as either original or translated. at inference time, they use their model in a zeroshot setting to translate from original source text into the distribution of original target text. similarly, @xcite label each source sentence with a binned qe score. by adding the label of the highest quality bin to a source sentence at inference time, they successfully bias the model towards high quality translations. ramos et al. (2024) apply rlhf (ziegler et al., 2020) to nmt using various qe metrics as reward, and compare it to data filtering, re-ranking using a qe model, and minimum bayes risk decoding (mbr) @xcite , finding that a combination of data filtering, rlhf, and re-ranking performs best. in dpo mbr fine-tuning, mbr is used to generate preference pairs for use with dpo @xcite . compared to dqo, this method is computationally more expensive, and requires a referencebased qe model. in addition, dqo's online nature ensures that preference pairs remain relevant to the policy model. @xcite apply rlhf with a reward model trained to distinguish high quality references (from literary translations) and translations sampled from their model. similar to us, they find evidence of cross-lingual transfer learning during preference learning. specifically, when optimized only on en-zh, their model improved for en to fr, es, ru, and ar. when training only on en-ar, however, they saw improvements in only half of the target languages. reward ranked fine-tuning (raft) is the method most similar to dqo, but uses sft to update the model towards a single preferred output rather than using dpo with a preferred/rejected output pair @xcite . as it was not evaluated for the translation task, used an independently trained reward model, and had slight differences in sampling parameters, we ran an ablation on whether to use dpo or sft in dqo (see section 4). ## conclusion we demonstrate the existence of a fundamental task-data mismatch in nmt and introduce direct quality optimization (dqo), a method of aligning pretrained models with human preference. using dqo on a multilingual nmt model, we find improvements in automatic quality metrics for all supported target languages, even those neither used for dqo, nor related to the languages used for dqo. a human evaluation confirms that these improvements reflect increased human preference. the improvements in translation quality for unrelated languages include language specific features that were not seen during dqo, suggesting that the baseline model had, but did not use, knowledge of those features during inference. we suggest that this is the expected behavior of a model trained with supervised learning, and present dqo as an efficient method of aligning a translation model with human preference. in an experiment on alma-13b-lora we confirm that dqo is applicable to decoder-only llms.