Cross-lingual Transfer of Reward Models in Multilingual Alignment (2025)

Abstract: Reinforcement learning with human feedback (RLHF) is shown to largely benefit from precise reward models (RMs). However, recent studies in reward modeling schemes are skewed towards English, limiting the applicability of RLHF in multilingual alignments. In this work, we investigate the cross-lingual transfer of RMs trained in diverse languages, primarily from English. Our experimental results demonstrate the strong cross-lingual transfer of English RMs, exceeding target language RMs by 3~4% average increase in Multilingual RewardBench. Furthermore, we analyze the cross-lingual transfer of RMs through the representation shifts. Finally, we perform multilingual alignment to exemplify how cross-lingual transfer in RM propagates to enhanced multilingual instruction-following capability.

## introduction recent advances in reinforcement learning with human feedback (rlhf) as a large language model (llm) post-training technique @xcite highlight the importance of having high-quality data @xcite and reward model (rm) @xcite @xcite . leveraging synthetic data has contributed to building stronger english rms due to their efficiency and scalability @xcite @xcite . nevertheless, adopting rms for non-english languages is heavily understudied. while llm-as-a-judge can be used as a generative reward model for multilingual rlhf settings @xcite , generative rms have been shown to underperform traditional rms @xcite . meanwhile, @xcite empirically demonstrates the possibilities of cross-lingual transfer in rms, but the findings were limited to simple tasks and encoder-decoder models. in this paper, we show that rms trained on english-only datasets (i.e., english rms) display strong cross-lingual transfer when built on top of multilingual pre-trained language models (mlms). we first demonstrate the cross-lingual transfer of english rms by consistently outperforming target language rms in multilingual rewardbench. then, we explain it with two reasons: 1) english preserves representations of the initial mlms (section 3.1), and 2) representations of mlms inherently have a strong understanding of languages (section 3.2), concluding that rms should preserve representations of mlms for generalizability. additional analysis of off-the-shelf rms supports our findings by both classifier and generative rms based on mlms having strong cross-lingual transfer. finally, multilingual alignment experiments exhibit the propagation of strong cross-lingual transfer in english rms to downstream usage, having an average win rate increase of 9.5% across four non-english languages. we empirically verify the cross-lingual transfer in reward models (rms) trained with different languages, thereby showing that the english preference data is a lingua franca in reward modeling. ## analysis on lingual tranfer of mlm this section provides empirical and theoretical insights on why english is lingua franca in reward modeling, given a multilingual language model (mlm) using two arguments: 1) english acts as a lingua franca in reward modeling because it best preserves the representations of the base model, and 2) representations in mlms should be preserved since they are inherently effective in language-aware encoding. ## multilingual alignment using rm in this section, we perform experiments to outline the effects of using the reward models (rms) from section 2 and how their cross-lingual transfer can propagate to the actual alignment process. ## cross-lingual transfer of external rms along with the controlled comparisons in section 2, we analyze the cross-lingual transfer in off-theshelf models on the original rewardbench through multilingual rewardbench. to ensure diversity in reward modeling schemes, we selected two classifier reward models (rms), armorm-8b @xcite and offsetbias-8b @xcite , alongside two generative rms, gpt-4o foot_1 and self-taught-llama-70b @xcite . 1 . classifier rms two classifier rms are both trained on top of llama3-8b-instruct @xcite , which are based on multilingual pre-trained language models (mlms) as discussed in appendix c. as in generative reward models interestingly, we can observe strong cross-lingual transfer in the generative rms in