Direct Preference Optimization for Neural Machine Translation with Minimum B ayes Risk Decoding (2024)

Abstract: Minimum Bayes Risk (MBR) decoding can significantly improve translation performance of Multilingual Large Language Models (MLLMs). However, MBR decoding is computationally expensive. We show how the recently developed Reinforcement Learning technique, Direct Preference Optimization (DPO), can fine-tune MLLMs to get the gains of MBR without any additional computation in inference. Our method uses only a small monolingual fine-tuning set and yields significantly improved performance on multiple NMT test sets compared to MLLMs without DPO.

## introduction mbr decoding @xcite @xcite is a twopass procedure that generates multiple translation hypotheses and selects a hypothesis based on bayesian risk. recent work @xcite @xcite has shown that mbr decoding can significantly boost the translation performance of mllms @xcite @xcite , outperforming greedy decoding and beam search. however, mbr decoding is expensive, both in computation and in latency. our goal is to fine-tune a base mllm so that it has the same single-pass decoding performance as mbr decoding. we propose a novel self-supervised fine-tuning method based on dpo @xcite . our method uses mbr decoding on an mllm to produce a preference dataset consisting of pairs of ranked translations. the dpo algorithm is used to fine-tune the mllm to prefer the higher-ranked translations over lower-ranked ones. mllms optimized for mbr preference achieve significantly better translation performance when decoded with beam search, achieving translation quality on par with mbr decoding of the original model. ## mbr and dpo we follow the expectation-by-sampling approach to mbr @xcite . given a set of sampled translations h(x) = {y ′ ∼ p (•|x)} and a loss (or utility) function l(•, •), the score (negative bayes risk) of each translation is found as this is simple but expensive. our goal is to train a model that produces translations with scores consistent with mbr, but without multi-step decoding. ## methodology our method combines mbr decoding and dpo fine-tuning @xcite . we use the mbr procedure to calculate a score (equation 1 ) for each of a set of translation hypothesis generated by the base model. we then fine-tune the base model using the dpo objective (equations 4,5) where the winning and losing hypotheses provided to dpo are chosen based on their relative mbr scores. if successful, the fine-tuned model will have learned to rank translations consistently with mbr decoding under the base model. ## dpo mbr fine-tuning and mt datasets: we evaluate translation on the wmt21 news translation test sets @xcite and the wmt22 general translation for chinese-english @xcite , and the iwslt 2017 test set for french-english @xcite . for dpo fine-tuning we use the source language text in the wmt20 test sets for chinese-english @xcite and iwslt 2017 validation sets for french-english. we do not use the corresponding reference translations, as dpo mbr fine-tuning is unsupervised. the fine-tuning and test sets are distinct and do not overlap. models: we use the bloomz and bloomz-mt models @xcite with 7.1 billion parameters as our base model. bloomz-mt was pre-trained on 366 billion tokens from monolingual texts and was fine-tuned for translation task on flores-200 @xcite and tatoeba @xcite datasets. to prompt the model for translation, we include two randomly selected translation examples from the fine-tuning set into the input prompt as demonstration examples; these prompts are kept fixed throughout. in addition, we also fine-tuned the bloomz-mt model in a supervised fashion for each language pair and denote this third base model as bloomz-mt-sft. we use previous wmt news translation test sets from 2017 to 2020 as supervised fine-tuning sets for chinese-english, and the first 20000 translation pairs from the iwslt 2017 training set for french-english. training details can be found in appendix b. evaluation metrics: we use three evaluation metrics: bleu @xcite , bleurt (sellam et al., 2020), and comet-22 @xcite . bleu serves only as a safety check: ideally dpo fine-tuning should not decrease bleu. baselines and targets: we take the base model and evaluate it on all the test sets with both beam search and mbr decoding. our fine-tuned models, when decoded with beam search, should achieve similar performance as mbr decoding under the base model and show improvement over the base model. we investigate two questions: ## conclusion we introduce dpo mbr fine-tuning, an unsupervised preference optimization algorithm that leverages the ranked lists from mbr decoding to teach mllms the preference of mbr decoding. our method enables mllms to achieve significant performance improvement when decoded with beam search in one pass, on par with the performance gained from two-pass mbr decoding foot_0 .