Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment (2024)

Abstract: Aligning language models (LMs) based on human-annotated preference data is a crucial step in obtaining practical and performant LM-based systems. However, multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages. In this work, we evaluate a simple approach for zero-shot cross-lingual alignment, where a reward model is trained on preference data in one source language and directly applied to other target languages. On summarization and open-ended dialog generation, we show that this method is consistently successful under comprehensive evaluation settings, including human evaluation: cross-lingually aligned models are preferred by humans over unaligned models on up to >70% of evaluation instances. We moreover find that a different-language reward model sometimes yields better aligned models than a same-language reward model. We also identify best practices when there is no language-specific data for even supervised finetuning, another component in alignment.

## introduction alignment has become an indispensable stage for building practical language models (lms) adjusted to human preferences. this additional step, however, makes it challenging to develop lms for many languages: unlike for autoregressive language modeling where multilingual unlabeled data may be easy to obtain @xcite , such as religious texts @xcite , labeled preference data can be expensive to gather. how do we align a lm in a target language without any preference data in that language? we propose a novel reward model (rm) transfer setup, where we re-purpose a trained rm for some source language to align a lm in a target language (figure 1 ), and investigate the effectiveness work done while zw was a part-time intern at google. to align in a target language (in this example, spanish), common monolingual alignment uses a rm for that target language. instead, we re-purpose a rm for a different source language (in this example, english). figure 2 : performing target-language alignment using a rm for a different source language improves performance, when evaluated exclusively in the target language. this improvement is sometimes even larger than using the target-language rm (monolingual alignment). here we measure the win rate against the target-language (unaligned) sft model judged by humans, and the 95% confidence interval across validation instances. "@xmath0. our results show that rm signals are generalizable and robust to input distribution changes, which could be leveraged for more future applications. practically, our findings pave the path towards lowering the costs for training and deploying lms that more equitably serve users around the world. ## background: alignment from human feedback in addition to traditional unsupervised lm pretraining, many recent lms also include an alignment phase to improve helpfulness, harmlessness, etc., supervised by human feedback @xcite @xcite . a common recipe includes three stages: supervised finetuning (sft), reward modeling (rm), and reward optimization. we give an overview of each and refer readers to @xcite and @xcite for details. we assume a base model already pretrained using a usually next-token prediction objective. the sft stage initializes from the base model and takes task inputs @xmath1. specifically, it optimizes the conditional log-likelihood of y given some input x, similar to regular language modeling. we denote the trained sft model using π sft . the rm stage trains a model r : x × @xmath2. it initializes from π sft and is trained using a dataset of human judgments of generations. we consider two types of feedback to train the rm: ## reward model transfer for cross-lingual alignment the pipeline in §2 is usually performed monolingually, commonly in english. aligning for a new language requires both sft data and rm data in that language. while the former may be relatively easier to obtain due to automatic construction methods, such as by re-purposing existing multilingual datasets @xcite or by eliciting from lms @xcite , rm data for a new language can be more expensive to gather, as it in principle requires human judgments. additionally, rm data should ideally be periodically re-collected to avoid over-optimization @xcite , further increasing data demand. thus, we are mainly interested in alignment without targetlanguage rm data, though, in §5.3, we investigate dispensing with target-language sft data too. we propose to perform reward optimization using a rm trained for a different language (figure 1 ). intuitively, assuming model generation quality transfers cross-lingually (e.g., good english generations are still good when translated into spanish foot_0 ), a model that can judge the output quality in one language should generalize to others, as long as the rm understands the languages, which is enabled by multilingual base model training. this generalizability is often observed for other tasks in the zero-shot cross-lingual transfer literature @xcite @xcite @xcite , and we expect it to work for rms too. a simple baseline would be to use automatically translated rm data, to which we compare in §5.1. in this paper, we use source language to denote the rm language, and target language for the language of the aligned model. ## experimental setup we consider two tasks: summarization, common in alignment research @xcite @xcite , and open-ended dialog generation, with substantial real-world relevance. §a describes dataset details and statistics. summarization. the seahorse dataset @xcite ) contains documents and summaries in six languages (german, english, spanish, russian, turkish, and vietnamese) with pointwise human ratings which we use. for sft, we gather the data sources of seahorse: xsum @xcite , xl-sum @xcite , mlsum @xcite , and wikilingua @xcite . we use mt5-xl @xcite as our multilingual base model, with 3.7b parameters. open-ended dialog generation. we use the openassistant dataset @xcite with multilingual, pairwise human-rated chat transcripts. 2 for the sft data, we use the human-preferred response in each pair to finetune the model. many languages in openassistant have only limited data, so we only consider three languages with the most amounts of data: english, spanish, and russian. we use palm-2-xxs as the base model @xcite . the authors of openassistant found rl to be ineffective for this dataset @xcite , which we confirmed in our experiments (figure 4 ). we therefore focus on best-of-n for this task. evaluation. we assess model quality across several settings. first, we use the target-language rm, which is by design finetuned to judge targetlanguage generation quality. but because of potential rm biases @xcite @xcite , we also include two zeroshot-prompted evaluation models with much larger backbones-gpt-4 (openai, 2023) and palm-2-l @xcite . this latter evaluation setup is common in prior work and has been demonstrated to correlate well with human judgments @xcite @xcite @xcite . we also confirm its validity in §5.1 and §c. importantly, both evaluation lms support multilingual texts. finally, we also perform human evaluations by self-reported native or advanced speakers, though only for a subset of language pairs and 250 (rl) / 100 (best-of-n) instances per pair due to its cost. for both human and lm evaluation, we elicit pairwise judgments to compare responses from the aligned model and the sft model @xcite @xcite . we measure the win rate, i.e., how often the judge prefers the former. a 50% win rate indicates no improvement from alignment. §g.2 includes more details such as the evaluation prompts and positional bias control. ## results here we report the results of cross-lingual alignment. see §h for numerical results that correspond to the plots in this section. ## analysis the effectiveness of cross-lingual alignment motivates us to better understand how it relates to various factors. we show that while rm generalizability within the original reward modeling task is a prerequisite, it does not uniquely explain the downstream success. similarly, we also show that the pairwise win rates (judged by palm-2-l unless otherwise mentioned) cannot be fully explained by, and thereby not predictable from, language features or the kl-divergence from the sft model. ## conclusion we showed through two different tasks that we can perform alignment using a different-language rm. surprisingly, we find this to be sometimes more effective than using a same-language rm. we also identified issues and remedies when we dispense with target-language sft data. we hope our findings can motivate future work to build better lms for more languages. adapting our rm transfer setup to other settings such as domain generalization would also be exciting future directions.