Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment (2025)

Abstract: Direct Preference Optimization (DPO) has become a prominent method for aligning Large Language Models (LLMs) with human preferences. While DPO has enabled significant progress in aligning English LLMs, multilingual preference alignment is hampered by data scarcity. To address this, we propose a novel approach that \textit{captures} learned preferences from well-aligned English models by implicit rewards and \textit{transfers} them to other languages through iterative training. Specifically, we derive an implicit reward model from the logits of an English DPO-aligned model and its corresponding reference model. This reward model is then leveraged to annotate preference relations in cross-lingual instruction-following pairs, using English instructions to evaluate multilingual responses. The annotated data is subsequently used for multilingual DPO fine-tuning, facilitating preference knowledge transfer from English to other languages. Fine-tuning Llama3 for two iterations resulted in a 12.72% average improvement in Win Rate and a 5.97% increase in Length Control Win Rate across all training languages on the X-AlpacaEval leaderboard. Our findings demonstrate that leveraging existing English-aligned models can enable efficient and effective multilingual preference alignment, significantly reducing the need for extensive multilingual preference data.

## introduction direct alignment algorithms (daas), such as dpo @xcite and its variants @xcite @xcite , renowned for their simplicity, efficiency and stability than reinforcement learning from human feedback (rlhf) @xcite ), * corresponding author original model iteration 1 iteration 2 win rate on x-aplacaeval 17.4 26.4 31.4 12.4 19.5 23.4 13.8 22.5 25.8 11.9 19.7 26.3 13.1 21.2 25.4 en es ru de fr figure 1: iterative preference transfer and improvement with implicit cross-lingual rewarding based on the english-aligned llama3 model. detailed results are shown in this work explores a novel perspective: leveraging the preference knowledge embedded within existing english-aligned models to facilitate multilingual preference alignment. prior work @xcite ... 2024a) has demonstrated that the implicit reward model, derived from the logits of a well-aligned english dpo model and its reference model, effectively captures preferences over english instructions. building on this, we apply this implicit reward model to the multilingual setting, using it to label preference relations in cross-lingual instructionresponse pairs @xcite . this ensures that multilingual responses are evaluated based on their alignment with english instructions. we term implicit cross-lingual rewarding, which preserves reward signal fidelity by directly evaluating multilingual responses under english instructions, avoiding translation-induced distortions. as shown in figure 2 , our approach involves three key steps: (1) multilingual response generation: starting from any multilingual model that is dpo-tuned on english preference data from an initial reference model. responses are sampled by the model from multilingual prompts. (2) implicit cross-lingual rewarding: constructing crosslingual instruction-response pairs by pairing english instructions with sampled multilingual responses. the implicit reward model then assigns preference scores to these responses, capturing the model's learned preference knowledge. (3) preference transfer training: our approach adopts iterative dpo similar to previous works @xcite , incorporating a negative log-likelihood (nll) loss term to train on the multilingual preference data, thereby transferring preferences across languages. our experiments start with the existing englishaligned llama 3 model, followed by two iterations of our training process. results (figure 1 ) demon-strate that our approach not only transfers preference knowledge from english to other languages but also iteratively improves english alignment through implicit reward. this suggests that each iteration inherently facilitates both preference transfer and refinement within the multilingual llm. notably, experiments with other daas-aligned base models and lower-resource languages confirm the broad applicability of implicit cross-lingual rewarding, establishing it as an efficient and robust strategy for enhancing multilingual preference alignment for any english-aligned model. ## preliminaries this section introduces two prominent methods in preference optimization, reinforcement learning with human feedback (rlhf) and direct preference optimization (dpo), and derives the implicit rewards of the dpo-tuned model. in preference optimization, the preference data typically takes the pairwise form, denoted as @xmath0, y w , y l )}. each prompt x is paired with two possible responses, y w and y l , where y w is designated as the preferred response by human evaluators. ## discussion in this section, we explore two key questions: 1. are there alternative forms of implicit reward? 2. is cross-lingual reward effective? ## conclusion this paper proposes a simple yet effective framework that leverages the implicit reward model of english-aligned models as a fine-grained reward signal to bootstrap multilingual llm alignment through a self-improving process. our key insight is to directly leverage english-aligned models and introduce an implicit cross-lingual reward mechanism to generate preference labels, thereby explicitly capturing preference knowledge from aligned model. this labeled preference data is then used to fine-tune the model itself via direct alignment algorithms, enabling the transfer and refinement of preferences from english to other languages. experimental results based on llama3 demonstrate that our approach significantly improves multilingual preference alignment without any annotation data. this work offers a novel and efficient pathway for multilingual preference alignment.