A Preference-driven Paradigm for Enhanced Translation with Large Language Models (2024)

Abstract: Recent research has shown that large language models (LLMs) can achieve remarkable translation performance through supervised fine-tuning (SFT) using only a small amount of parallel data. However, SFT simply instructs the model to imitate the reference translations at the token level, making it vulnerable to the noise present in the references. Hence, the assistance from SFT often reaches a plateau once the LLMs have achieved a certain level of translation capability, and further increasing the size of parallel data does not provide additional benefits. To overcome this plateau associated with imitation-based SFT, we propose a preference-based approach built upon the Plackett-Luce model. The objective is to steer LLMs towards a more nuanced understanding of translation preferences from a holistic view, while also being more resilient in the absence of gold translations. We further build a dataset named MAPLE to verify the effectiveness of our approach, which includes multiple translations of varying quality for each source sentence. Extensive experiments demonstrate the superiority of our approach in “breaking the plateau” across diverse LLMs and test settings. Our in-depth analysis underscores the pivotal role of diverse translations and accurate preference scores in the success of our approach.

## introduction the emergence of large language models (llms) has significantly transformed the landscape of nlp, showcasing outstanding capabilities in a spectrum of nlp tasks @xcite @xcite . this transformation extends to machine translation (mt) @xcite @xcite . through supervised finetuning (sft) using a small amount of parallel data, llms demonstrate the capability to compete with established commercial translation services such as google translate, particularly in high-resource languages @xcite . nevertheless, sft trains the model to imitate reference translations token by token, making it vulnerable to the noise present within the data @xcite @xcite . the noise can stem not only from the lack of attention by annotators, but also from the inherent challenge of achieving perfect translations due to the intricate interplay of language, culture, and vocabulary. as an adept translator requires not only linguistic proficiency but also a deep understanding of cultural contexts and nuances in both the source and target, it is nearly unattainable to gather extensive parallel translations of top-notch quality @xcite @xcite . as a result, the performance enhancement achieved through sft often quickly reaches a plateau. further increasing the volume of parallel translations typically yields minimal additional benefits, and may instead impair the translation capabilities of llms @xcite . to alleviate aforementioned limitation of sft, endeavors have been made to provide llms with holistic assessment of contrasting examples rather than token-level imitations. @xcite ; @xcite add a flawed translation to the reference translation in the model input, encouraging the target llm to recognize their quality difference. @xcite also use a pair of translations, but they additionally optimize the llm to favor better translations through ranking loss. nevertheless, these works have shared limitations. first, the flawed translations are either generated by adding artificial noise to the reference translations or by other (smaller) mt systems. these imperfections in translations can be obvious and easy for llm to distinguish, weakening the learning signal. second, they only provide the relative ranking of the two translations, without quantifying the extent of their quality differences. in this work, we present a framework based on the plackett-luce model to explicitly align the generation probability of the target llm with human preferences @xcite . instead of using artificial noise, we collect contrasting translations generated by our target llm, directing our optimization efforts toward "hard negative examples" @xcite . human preferences are denoted with precise scores rather than general ranking orders to teach llms about the nuances in different translations. llms are then trained to enhance their capabilities incrementally from the learnt nuances without depending solely on the existence of "gold references", so as to effectively break the plateau associated with sft. we build a dataset, which we refer to as maple, to facilitate preference learning. it equips each source sentence with five translations in diverse quality, scored by professional translators. by performing preference learning on maple, our final mt model outperforms other mt models based on the same foundation llm by up to 3.96 comet score. we further show that while the intention of creating maple is to enhance our target llm, it can be reused to improve other llms, helping them break the performance plateau with up to 1.4m parallel data. finally, we analyze the key factors that make preference learning effective. our contributions are as follows. (1) we leverage preference learning to teach llms a holistic notion of translation quality. extensive experiments show that our model consistently outperforms strong baselines on two test sets across four translation directions. (2) we revisit the underlying modelling assumptions leading to the bradley-terry and plackett-luce ranking models and discuss how preference distances can be incorporated directly into the ranking models. (3) we meticulously construct an mt-oriented preference dataset, maple, employing professional human translators to obtain quality scores for multiple translations corresponding to the same source sentence. we release our dataset to facilitate future mt research. (4) our indepth analysis reveals that high-contrast pairs and accurate quality scores are crucial in enhancing the effectiveness of our approach, providing guidance for maximizing the benefits of preference learning. human preference alignment. @xcite align llms with human intentions and values by training a reward model for preference ranking and optimizing the llms through the ppo algorithm @xcite . however, the online reinforcement learning nature of ppo leads to considerable computational costs and is known for its sensitivity to hyperparameters @xcite . to ease the alignment, hu et al. (2023 @xcite to capture human preferences in alignment. in this work, we adopt the plackett-luce model to mt, teaching the model to discern nuances in different translations and to prefer accurate translations. ## methodology we aim to enhance llm in mt tasks via a twostage optimization process. we first fine-tune the target llm with a small set of high-quality parallel data to elicit its translation ability (section 3.1). this mirrors the supervised fine-tuning approach used in prior work, where llms were tailored to follow instructions @xcite . we then use preference learning to guide the llm to prioritize the generation of accurate translations over flawed ones (section 3.2). ## human preference data collection we build maple (machine translation dataset for preference learning), a dataset derived from wmt20/21 test sets. it contains multiple translations per source sentence, each assigned a realvalued human preference score. maple covers four translation directions: german-to-english (@xmath0, chinese-to-english (@xmath1, englishto-german (@xmath2, and english-to-chinese (@xmath3. for each direction, 1.1k source sentences are sampled from the test sets of wmt20/21. each source sentence is associated with five translations, including one reference translation from wmt20/21, and four translations generated by vi-cunamt, our target llm that we aim to improve through preference learning (see training details of vicunamt in section 5.1). among the four translations, one is generated using beam search with a beam size of four, and three translations are obtained through nucleus sampling @xcite with @xmath4.9. we also build a development set containing 200 source sentences per direction sourced from news crawl 2022. altogether, maple contains 5.2k source sentences and 26k translations with preference scores. see appendix b.1 for more detail on the translation collecting process. dataset statistics. the score distribution is shown in fig. 1 . the left side shows the score distribution by rank, and we can see maple contains translations that exhibit a wide range of qualities. the right side shows the score distribution by translation type, and as expected the reference is ranked highest, followed by the beam search and the nucleus samples. nonetheless, there is considerable overlap in the score distributions, and we find that in 21% of the cases, the beam search predictions are scored higher than the reference translation. ## experiments in this section, we present our mt model trained using the proposed two-stage framework and compare it with strong llm-based mt systems. datasets. we train and evaluate the model on data on four translation directions: en↔de and en↔zh. in the sft stage, we use high-quality test sets from wmt17/18/19 for training, containing 30k parallel sentences in total across the four directions. the wmt21 test set is used for validation. in the preference learning stage, we train on maple, and validation is done on the remaining data from wmt20/21 test sets which was not selected for inclusion in maple. we evaluate trained models on the test sets of wmt22 @xcite and @xcite . refer to appendix c.1 for detailed data statistics. training. in both sft and pl stages, we use a learning rate of 5e-6, an effective batch size of 96, and a linear learning rate schedule with a warmup ratio of 0.1. for each training instance, one mt instruction is randomly selected from an instruction pool containing 31 mt instructions. see appendix c.2 for the complete list of instructions. evaluation. at inference time, a fixed mt translation instruction is used. the maximum generation length is set to 512. we use a beam size of 4 for decoding and report bleu @xcite and comet @xcite scores. ## analysis reuse of preference data. maple contains the translations generated by vicunamt, which is also the target llm we aim to improve. there would be additional value if this data could be reused to improve other llms. to investigate this, we train both mistral-instruct and bloomz on maple using pl. as shown in limited gains with additional parallel data. section 5.2 shows that the maple dataset, which contains 4.4k preference examples, can be more valuable than an equivalent amount of parallel data with either the reference or the best translations. a natural follow-up question is whether adding more parallel data can close the gap. to answer this question, we collect more data by concatenating wmt20, wmt21 test data with news commentary v16, making 1.4m parallel sentences in total. 6 we fine-tune vicunamt and mistral-instructmt (i.e., mistral-instruct after sft stage) on different proportions of this data and plot the performance curve in figure 2 . in both cases, similar to observations in @xcite , adding more parallel data does not always improve these models and they never attain the performance level reached by using pl with maple. diverse translations help more. by default, we perform pl using all five translations provided by maple. we now study the relation between the final model performance and the number of preference translations used. we select @xmath5, 3, 4} translations and rerun the pl algorithm on vicu-namt and mistral-instructmt. we explore two selection modes for selecting k translations. given five translations sorted by human preference scores in descending order, the forward mode selects the first k translations (i.e., the best k), while the reverse mode select the first and last k -1 translations. we compare both modes varying k and present the results in figure 3 . there is a clear disparity in performance with these two selection modes. the reverse mode consistently outperforms the forward mode given the same number of translations, with a larger advantage in low-resource cases, such as when @xmath6. this is intuitive since the reverse mode always includes the highest-and lowest-scored translations and thus, pl may have a better chance to see "hard negatives" which have low human preference score but high generation probability. the general trend shows that including more preference samples is better, and using all available samples yields the best performance. ## conclusion we present a preference learning framework to break the performance plateau faced when performing sft. it enhances the translation capabilities of llms by motivating them to differentiate the nuances in different translations. to support this framework, we have carefully curated a preference dataset, named maple, featuring translations of varying quality, each scored by professional translators. extensive experiments, including human evaluations, confirm the effectiveness of this framework. in addition, we demonstrate that maple can be reused to enhance other llms, further bolstering its practical usability. future research could consider extending our framework into an iterative process for continuous improvement of llms' translation capabilities.