Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation (2024)

Abstract: Alignment with human preferences is an important step in developing accurate and safe large language models. This is no exception in machine translation (MT), where better handling of language nuances and context-specific variations leads to improved quality. However, preference data based on human feedback can be very expensive to obtain and curate at a large scale. Automatic metrics, on the other hand, can induce preferences, but they might not match human expectations perfectly. In this paper, we propose an approach that leverages the best of both worlds. We first collect sentence-level quality assessments from professional linguists on translations generated by multiple high-quality MT systems and evaluate the ability of current automatic metrics to recover these preferences. We then use this analysis to curate a new dataset, MT-Pref (metric induced translation preference) dataset, which comprises 18k instances covering 18 language directions, using texts sourced from multiple domains post-2022. We show that aligning TOWER models on MT-Pref significantly improves translation quality on WMT23 and FLORES benchmarks.

## introduction the use of large language models (llms) in machine translation (mt) has garnered significant attention from the research community @xcite . unlike traditional sequence-to-sequence mt models trained on parallel data @xcite , llm-based mt systems either use in-context learning to elicit translation knowledge acquired during pre-training @xcite or undergo supervised finetuning (sft) on high-quality translations to further enhance their translation capabilities @xcite @xcite . the default sft approach for llm-based mt is to tune systems based solely on single human reference translations. however, this kind of supervision might be insufficient to push quality further: first, because many valid translations may exist for a given source, with some preferred over others @xcite . second, because the next-token prediction objective of sft does not capture sentence-level semantics and quality criteria @xcite . this has motivated new approaches that go beyond sft to leverage translation preferences or quality feedback to improve learning @xcite @xcite . a key factor in aligning llms toward translation preferences is ensuring the quality and diversity of the datasets used for training @xcite @xcite . unfortunately, existing datasets have several limitations: first, they are created from translation outputs of one or two models, for limited language pairs, thereby restricting their diversity and applicability to novel scenarios. second, these datasets are either entirely automatically generated @xcite or completely based on human feedback @xcite . while automatic evaluation offers efficiency, it lacks the crucial validation that the metrics used truly align with human preferences. on the other hand, datasets that use human feedback, while high-quality and reliable, pose resource constraints and are challenging to scale. to bridge this gap, we provide a holistic approach to balance the advantages of automated metrics while ensuring that they lead to preferences that truly align with humans. we first collect sentence-level quality assessments and preferences from human expert translators ( §3)-we use the wmt23 english-german and chinese-english datasets @xcite with outputs from five high-quality mt systems: towerinstruct-7b, towerinstruct-13b @xcite ; alma-13b-r @xcite ; @xcite and googletranslate. using these assessments, we then examine the ability of automatic quality estimation (qe) metrics to recover human preferences. our findings show that an ensemble of xcomet-xl and xcomet-xxl @xcite )-xcomet-xl+xxlachieves the highest correlation with human judgments and a high precision score in identifying the preferred translations. using this analysis, we create a new mt preference dataset, mt-pref (metric-induced translation preference dataset), with source sentences mined post-2022 for 10 languages (english, german, chinese, russian, portuguese, italian, french, spanish, korean and dutch). translations for each source sentence are generated using diverse mt systems representing different architectures, training data, and quality levels ( §4). we use the ensemble metric xcomet-xl+xxl to get the most and least preferred translations from the set of hypotheses. experiments on aligning mtspecialized decoder-only models (tower) using existing preference learning algorithms with our mt-pref dataset demonstrate improved translation quality on the wmt23 @xcite and flores @xcite benchmarks, with larger gains in out-of-english translation directions ( §6). further analysis shows that the aligned models better rank translations according to human preferences over baselines. ## background: aligning mt with translation preferences given a source text, the goal of mt is to generate a translation that accurately reflects the information and meaning conveyed in the source. at training time, the mt model π θ goes through sft to minimize the negative log-likelihood (nll) loss induced by source-reference pairs (x, y): a drawback of sft is that it typically optimizes the model towards a single reference translation. in contrast, preference learning objectives incorporate relative preferences between alternatives, allowing the model to learn from subtle differences in translation quality @xcite . different variants of preference optimization (po) have been proposed in the literature. reinforcement learning from human feedback (rlhf) has shown to be effective in aligning model behavior with human values @xcite . @xcite propose direct preference optimization (dpo) as a simple and scalable alternative to rlhf. given a preference dataset d with source sentences x, preferred or chosen outputs y + and less preferred or rejected outputs y -, the model is trained with the following objective: one notable limitation of the dpo objective is that it requires both π θ and π ref in memory, significantly increasing memory requirements and computation costs. to address this, @xcite further approximate the dpo objective using a uniform reference model (π @xmath0: as the quality of the preference datasets used for training is key for its success @xcite @xcite , we next discuss our process of collecting a high-quality dataset for preference learning for mt. ## modeling user preferences via automatic metrics to create a high-quality preference dataset for mt, we need human judgments on translation outputs from strong mt systems. this helps us understand and model human preferences among competitive translations. since large-scale collection of these judgments is costly, we evaluate existing automatic metrics to see if they effectively reflect human preferences. this determines if metrics can be reliable proxies for human judgments when translation quality is high and preference variance is low. we describe the dataset, models, and task instructions given to the expert annotators used in our study in §3.1. the human evaluation results are presented in §3.2. finally, we discuss our metaevaluation of automatic mt metrics in their ability to recover human preferences in §3.3. ## mt-pref dataset building on the findings from §3, we create our preference dataset using xcomet-xl+xxl. we discuss the choice of the text and models in §4.1, followed by the method for inducing and selecting preference pairs from the dataset in §4.2. ## experimental settings we use the mt-pref dataset foot_5 to align mt models with translation preferences ( §4) and compare several preference learning methods detailed in §2. training data the mt-pref dataset contains 18k instances with approximately 1k examples for each translation direction. the counts of the chosen and the rejected hypotheses from each model and the distribution of metric scores are shown in fig. 2 . the nllb-54b model accounts for most of the rejected hypotheses (∼46%), whereas the chosen hypotheses are more equally distributed across the gpt-4, googletranslate, and the tower models, illustrating consistent and higher-quality translations generated by these models. evaluation we evaluate finetuned models on the wmt23 test set (en ↔ {de, ru, zh}) and the flores dev-test set (en ↔ de, ru, zh, es, fr, pt, nl, it, ko) using tower-eval @xcite . 8 we report system-level translation quality using chrf @xcite , comet, metricx-xxl @xcite , and xcomet-xl. we cluster system performance using the wilcoxon rank-sum test (@xmath1.05) with comet as the primary metric. rank ranges, denoted by [l+1, n-w], indicate the number of systems a particular system underperforms or outperforms, where l represents the number of losses, n is the total number of systems, and w is the number of systems that the system in question significantly outperforms @xcite . we compare the models' accuracy (% acc.) for selecting the best-over-worst hypothesis with the model's likelihood on the human preferences ( §3) after finetuning on mt-pref. ## results sft is necessary to obtain translation quality improvements using dpo. comparing different variants of dpo (dpo sft , dpo base and dpo base +sft), we find that either the sft phase or the sft regularization is necessary to obtain significant comet improvements. this also aligns with findings from @xcite who show that learning from chat preference datasets fails when skipping the initial sft stage. interestingly, dpo base attains the highest % acc. scores among variants, showing an improved ability to discern but not necessarily generate high-quality translations. we find that as suggested by @xcite , it is indeed because dpo base increases the relative probability between the two classes by decreasing the model's likelihood for both chosen and rejected translations (see fig. 3 ). ## related work llms for mt earlier works exploring llms to perform mt study prompting techniques to gener-ate translations @xcite @xcite with research focusing on selecting high-quality and relevant examples as demonstrations to incorporating external knowledge mimicking human-like translation strategies @xcite . more recently, several works have proposed finetuning llms to improve the translation quality @xcite , resulting in specialized models that attain competitive performance to state-of-the-art production level translation systems @xcite . across all methods, the quality of the data used for training is paramount to the finetuning methods. therefore, in this work, we focus on curating a high-quality translation preference dataset using metrics that closely reflect true human translation preferences and outputs generated from a diverse set of high-quality mt systems. quality feedback for mt using feedback from automatic metrics for mt or human quality assessment has been an active area of research through the past decade. this quality signal is either utilized during training @xcite @xcite @xcite @xcite or decoding @xcite @xcite or for modeling translation preferences in the dataset directly @xcite . similar to @xcite , we use automatic metrics to induce preferences in the dataset but with the additional validation that the chosen metric indeed reflects human quality expectations and with translations generated from diverse mt systems. ## conclusion we present mt-pref, a high-quality translation preference dataset, curated by combining the strengths of human evaluation and automatic metrics. the dataset includes metric-induced preferences from strong mt models across 18 language directions with new source sentences mined post-2022. aligning state-of-the-art decoder-only llms on this preference dataset using existing aligning tuning algorithms improves translation quality. furthermore, the aligned models are also better at modeling human preferences of translation quality.