Improving LLM s for Machine Translation Using Synthetic Preference Data (2025)

Abstract: Large language models have emerged as effective machine translation systems. In this paper, we explore how a general instruction-tuned large language model can be improved for machine translation using relatively few easily produced data resources. Using Slovene as a use case, we improve the GaMS-9B-Instruct model using Direct Preference Optimization (DPO) training on a programmatically curated and enhanced subset of a public dataset. As DPO requires pairs of quality-ranked instances, we generated its training dataset by translating English Wikipedia articles using two LLMs, GaMS-9B-Instruct and EuroLLM-9B-Instruct. We ranked the resulting translations based on heuristics coupled with automatic evaluation metrics such as COMET. The evaluation shows that our fine-tuned model outperforms both models involved in the dataset generation. In comparison to the baseline models, the fine-tuned model achieved a COMET score gain of around 0.04 and 0.02, respectively, on translating Wikipedia articles. It also more consistently avoids language and formatting errors.

## introduction decoder-only large language models (llms) serve as versatile tools for a variety of natural language processing tasks, such as question answering, summarization, and translation. typically, llms undergo three phases of training: pretraining, supervised fine-tuning (sft), and preference alignment. the quality of a translation depends on many fine details (e.g., style, semantics, figurative language, etc.), which might not be sufficiently well learned during sft. our hypothesis is that a model can improve on subtle differences between a reasonable and good translation through preference alignment. in this work, we present the training of an llm with direct preference optimization (dpo) @xcite for optimizing its translation abilities. using slovene as a use case, our primary goal is to develop a reliable open-source english to slovene translator that can be used for translating large english corpora to slovene. the main contributions of our research are: the approach is based on synthetic data generation and is suitable for less-resourced languages such as slovene. currently, there are no existing preference-annotated datasets for english to slovene translation. obtaining such a dataset using human translators and annotators would be slow and prohibitively expensive. hence, we automatically create such a dataset. our core insight is that even without access to a human-curated preference corpus for english to slovene translation, we can bootstrap a reliable preference dataset by exploiting the behavior of two independent llms and a suite of automated filters. by prompting both gams-9b-instruct @xcite and eurollm-9b-instruct @xcite to translate english wikipedia articles @xcite and a collection of english news articles from common crawl (cc-news dataset), we generate dual translations for each article, resulting in a dataset with around 67,000 entries from wikipedia and 30,000 from cc-news. whenever one model produces a clean slovene output and the other makes an obvious error -whether by continuing the conversation in the wrong language, by truncating the translation, or by adding unwanted prefixes -we can confidently mark the former as chosen and the latter as rejected. to capture finer quality differences, we score all translations without any obvious mistakes with comet @xcite and select pairs whose comet scores differ by some minimum threshold. the result is a diverse, synthetic preference dataset that reflects both unacceptable errors (wrong language, incomplete output, etc.) and subtler language fluency distinctions captured by the comet scores. after curating the translation pairs, our final preference dataset consists of around 25,000 entries for wikipedia and 10,000 from cc-news. using generated synthetic preference data, we apply direct preference optimization (dpo) to the gams-9b-instruct model. we chose dpo because it directly optimizes the likelihood ratio between chosen and rejected outputs, sidestepping the instability and rewardmodeling overhead of standard reinforcement learning from human feedback (rlhf) pipelines @xcite . we train the model on the generated preference pairs over three epochs on 16 a100 gpus. we employ a linear learning-rate warm-up followed by cosine decay to stabilize the early dpo updates. this fine-tuning framework takes advantage of both our filtering heuristics and dpo's principled ranking objective, driving the 9b-parameter model toward more fluent and complete slovene translations. we evaluate the effectiveness of our approach on the public slobench leaderboard 4 and on unseen wikipedia and cc-news articles. we show that our dpo-trained gams-9b-instruct model outperforms the original model variant on slobench and almost matches the performance of considerably larger gams-27b-instruct @xcite . additionally, on our wikipedia benchmark, testing for language and formatting consistency, our model achieves an error rate of 0.8 %, which is a substantial improvement over the original model's 12 %. the rest of the paper is organized into five sections. in section 2, we present related work on the development of slovene language models and other tools we rely on. in section 3, we explain how the training data was generated through our heuristic approach and automatic metrics. the training pipeline and hyperparameter search are presented in section 4. in section 5, we evaluate our model and compare it to other state-of-the-art open-source models. we provide conclusions, limitations, and directions for further work in section 6. ## background and related work our work is related to other research on llms for less-resourced languages, like slovene, the ongoing research in preference alignment of llms, machine translation, and evaluation methods for machine translation. these topics are outlined below. ## synthetic preference data generation obtaining a high-quality preference annotated translation dataset is challenging. we take english wikipedia and news articles from common crawl as a starting point, as those cover a wide variety of topics. our data generation pipeline consists of multiple stages. we start by generating translations, described in section 3.1. this is followed by identifying trivial errors, described in section 3.2, and scoring remaining instances, described in section 3.3. our final data construction is described in section 3.4. ## translator training to produce an improved llm for mt, we take gams-9b-instruct model as our starting point and optimize it for translation on the preference dataset from section 3 using the dpo method. we provide a brief description of the method in section 4.1. we describe our training implementation in section 4.2. in section 4.3, we describe the hyperparameter search performed. ## evaluation and results we evaluate our model on two benchmarks. the first is a public slovene-to-english benchmark, that is part of the public benchmark suite slobench. since slobench evaluates the models only on persentence translations, we also evaluate our model on a custom evaluation set relevant to our specific goal of translating longer english documents for training large language models for slovene. we create such an evaluation set based on a set of unseen english wikipedia and cc-news articles. throughout this section, we refer to our model as gams-9b-dpo-translator. ## conclusion we proposed a pipeline for improving machine translation based on data generation and dpo preference alignment method. we showed that our approach increases the quality of the trained model's translations. we showed a small performance improvement on the slobench evaluation, and a substantial improvement in translating longer documents, such as wikipedia articles. the main goal of our research was to make an open-source translator for a less-resourced language (slovene) more reliable. since our approach is language agnostic (given the precondition of the existence of at least two machine translation options for this language), it can be applied to many less-resourced languages or specific domains. we believe that our approach will help translate high-quality english corpora to less-resourced languages which is necessary to build llms in such languages and important for the sovereignty of such languages in the llm era. we plan to use the insight gained during this project to fine-tune the 27b parameter model with the same training pipeline. since the systems are already in place, the remaining challenges are to scale the training process and to obtain the required computational resources. scaling for a larger model would involve generating more training data and using more advanced distributed training optimizations such as zero stage 3, zero++ @xcite . additionally, the recently released nvidia nemo-rl framework foot_2 shall be tested. a potential improvement to consider in the future is curriculum dpo @xcite instead of vanilla dpo. curriculum learning would allow the model to learn on different datasets, step-by-step, increasing in difficulty. the datasets could be divided into two major groups. the first group would contain the training examples from our heuristic-based analysis (language and truncation errors), and the second group would have the training examples ranked by comet score. the latter could be further subdivided into multiple subsets based on the comet score difference between chosen and rejected responses. a lower score delta indicates a more subtle difference in quality, and the model would be trained on those after it had been trained on the pairs with a more obvious quality difference. finally, other preference alignment methods, such as grpo, could be tested. since we have already automated response rankings, we would have to turn those rankings into a reward function for grpo. another possibility would be combining both methods by first focusing on language and truncation errors using dpo and then performing grpo based on comet scores.