RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLM s (2024)

Abstract: Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on a small set of high-resource languages like English and Chinese. This captures a small fraction of the languages in the world, but also makes it unclear which aspects of current state-of-the-art research transfer to a multilingual setting. In this work, we perform an exhaustive study to achieve a new state of the art in aligning multilingual LLMs. We introduce a novel, scalable method for generating high-quality multilingual feedback data to balance data coverage. We establish the benefits of cross-lingual transfer and increased dataset size in preference training. Our preference-trained model achieves a 54.4% win-rate against Aya 23 8B, the current state-of-the-art multilingual LLM in its parameter class, and a 69.5% win-rate or higher against widely used models like Gemma, Mistral and Llama 3. As a result of our efforts, we expand the frontier of alignment techniques to 23 languages, covering approximately half of the world’s population.

## introduction what languages are favored in technological progress is often deeply intertwined with historical patterns of technology access and resources @xcite @xcite . preference optimization is a valuable and widely adopted post-training technique to align large language models (llms) with human preferences @xcite @xcite . it has also been shown to lead to large gains in performance across a wide variety of nlp tasks @xcite @xcite . to-date, the majority of progress in preference optimization has over-fit to a small handful of languages, resulting in large gaps in performance outside of english @xcite @xcite @xcite , and also risks introducing security flaws for all users @xcite @xcite @xcite . while expanding the frontier of what languages are supported by ai is an increasingly urgent problem, extending preference optimization to a multilingual setting is a non-trivial challenge. first, numerous works have shown that multilingual modeling typically faces both a data scarcity and data quality problem @xcite @xcite @xcite . this is even more pronounced for high-quality preference data which is virtually non-existent in many languages. collecting multilingual preference datasets through human annotation is expensive and time intensive @xcite , and while prior works have proposed the use of llms to synthetically create preference datasets @xcite @xcite , these efforts predominantly focus on english. the few efforts that have focused on multilinguality have relied on translation, introducing artifacts and resulting in a lack of diversity in preference pairs @xcite , which is known to be critical to model performance @xcite @xcite @xcite . second, preference optimization in many languages simultaneously is a difficult machine learning task. the lack of studies in preference optimization outside of english raises questions on how findings from monolingual optimization would transfer. training dynamics of rlhf are known to be often unstable @xcite @xcite , which can be exacerbated by the involvement of multiple languages where preference data is from a heterogeneous distribution and negative transfer between languages is possible @xcite . the few existing works on multilingual rlhf @xcite exhibit poor results and are outperformed by massively multilingual language models without any preference optimization @xcite . poor performance of the few existing multilingual preference optimization works begs the question: is this a result of fundamental limitations with standard preference optimization techniques (especially in heterogeneous optimization settings like multilingual ) or whether we are lacking high quality multilingual data? in this work, we exhaustively study the aforementioned challenges. our goal is to systematically understand key variables which might impact multilingual alignment, including the source and amount of available preference data, offline vs online rlhf techniques, and the effect varying number of languages covered in preference optimization training data. we complete a comprehensive set of experiments with state-of-the-art alignment techniques including dpo @xcite and rloo @xcite starting from the 8-billion-parameter instruction-finetuned aya model covering 23 languages (aya-23-8b; @xcite . our findings can be summarized as follows: ## experimental set-up multilingual base model we perform all experiments with aya 23 8b @xcite which is chosen because it (1) is massively multilingual, pre-trained and supervised fine-tuned for 23 languages, and (2) it achieves state-ofthe-art multilingual performance in 23 languages compared to other commonly used llms in its class, outperforming mistral-7b @xcite , gemma-7b (gemma-team, 2024), and aya-101-13b @xcite . on multilingual benchmarks and open-ended generations, aya 23 achieves a 65% win-rate or higher in head-tohead comparisons with popular open sourcee models @xcite . furthermore, aya 23 is an open weights model that is not preferencetrained, allowing us to isolate the impact of multilingual preference optimization. reward model we use a closed-source multilingual reward model (rm) which is competitive with top-scoring state-of-the-art rms on the reward-bench leaderboard @xcite . this reward model achieves high llm response ranking agreement with gpt-4-turbo on english and non-english languages as shown in preference optimization training we train aya 23 8b model for 2 epochs in both dpo and rloo experiments. dpo runs are trained with kl-penalty β = 0.5, learning rate 5e-7, and adamw optimizer (kingma and ba, 2014). rloo runs are trained with rloo @xmath0, klpenalty β = 0.01, learning rate 5e-6, adamw optimizer, and online generation sampling temperature of 0.75. all runs are performed on a single node with either 8 x nvidia a100 80gb or 8 x nvidia h100 80gb gpus with deepspeed zero stage 3 (rajbhandari et al., 2020) and full fine-tuning of all 8 billion model parameters. we performed hyperparameter sweeps for learning rate @xmath1, 5e-7, 5e-6, 5e-5} and for klpenalty β ∈ {0.05, 0.1, 0.5} for both dpo and rloo to the best of our ability. for a fair comparison with dpo, we utilize the same reward model for rloo which is used to generate our synthetic multilingual preference dataset (for ranking the generations). ## results and discussion win-rates against open-weights models figure 1 and win rates against aya-23-8b ## conclusion our work presents a comprehensive study on multilingual preference optimization. we show that the inclusion of multilingual data in preference optimization leads to significant improvements in multilingual performance over english-only preference optimization. this improvement scales both with the number of languages and the total number of examples included in the training data. additionally, we show that preference training exhibits cross-lingual transfer, leading to significant gains in languages not present in the training data. we also find that using online preference optimization outperforms offline preference optimization, highlighting the importance of online samples during training. as a result of our study, we expand the frontier of alignment techniques to 23 languages which cover half the world's population, by successfully preference-training an 8 billion aya 23 model that outperforms both the original aya 23 8b base and widely used models including gemma, mistral, and llama 3.