Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback (2023)

Abstract: A key technology for large language models (LLMs) involves instruction tuning that helps align the models’ responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are applied to produce the best commercial LLMs. To improve the accessibility of LLMs, various instruction-tuned open-source LLMs have also been introduced recently. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their accessibility to many other languages in the world. In addition, SFT has been used as the only approach to instruction-tune open-source LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework with created resources, fine-tuned LLMs, interaction scripts are released at https://github.com/nlp-uoregon/Okapi. A demo video to show our framework can also be found at: https://youtu.be/QFV2fkPwvi0.

## introduction pre-trained on massive data, large language models (llms) with hundreds of billions of parameters such as gpt-3 @xcite can unlock new emergent abilities that cannot be achieved with smaller models @xcite @xcite . however, as llms are trained with the autoregressive learning objective, they might exhibit unintended behaviours from human expectations @xcite @xcite . to overcome this issue, instruction fine-tuning has been proposed as a prominent approach to improve capabilities in following human instructions for llms and align them with human intentions in conversations @xcite @xcite . as such, two major techniques for instruction tuning feature supervised fine-tuning (sft) and reinforcement learning from human feedback (rlhf) that are leveraged by the best commercial llms such as chatgpt and gpt-4 to deliver outstanding dialog performance. another issue with llms pertains to the massive scales and closed-source nature of the commercial llms that greatly restrict accessibility and the extent of interactions with the technology. to this end, there have been growing efforts from the open-source community to create more accessible llms with affordable scales while securing competitive performance as the proprietary llms, e.g., llama @xcite , stablelm @xcite , falcon @xcite , and mtp @xcite . instruction tuning has also been applied to these open-source llms to improve their abilities to engage with human, and different instruction datasets have been collected to facilitate the process, e.g., alpaca @xcite , vicuna @xcite , lamini-lm @xcite , and dolly @xcite . however, the instruction-following abilities of existing open-source llms have been developed mainly for english and some popular languages (i.e., using instruction data for those languages), failing to support many other languages of the world to serve a broader population @xcite . to overcome this challenge, a few contemporary frameworks have explored instruction tuning of llms for multiple languages, i.e., phoenix @xcite and bactrian-x @xcite . however, their multilingual instruction tuning efforts are limited to only supervised fine-tuning, which is unable to examine reinforcement learning with human feedback (rlhf) to further boost the performance for multilingual llms. to fill in this gap, our work aims to develop okapi, an open-source framework with rlhfbased instruction-tuned llms for multiple languages to provide resources and shed light on their performance for multilingual llm learning. okapi will emphasize on less studied languages and opensource llms to better democratize the benefits of instruction-tuned llms. in particular, an example in the instruction datasets involves an instruction, an input text, and a desired response output/demonstration. in sft, the pre-trained llms are fine-tuned over the instruction triples (instruction, input, output) via supervised learning to promote their alignment with human expectations. in rlhf, generated outputs from the sft-tuned llms are first ranked to provide training signals for the reward functions. afterward, the sft-tuned models will be further optimized via reinforcement learning utilizing rewards from the trained reward models. as such, rlhf has been successfully employed to create effective commercial llms (e.g., instructgpt, chatgpt), owning to its ability to learn beyond positive examples associated with only desired demonstrations. by leveraging the reward models, rlhf can observe lower ranking scores for less accurate demonstrations to obtain richer training signals for llms. to our knowledge, okapi is the first work to perform instruction tuning with rlhf for open-source llms over multiple languages. to develop okapi, we need to overcome the scarcity of instruction datasets in multiple languages to train and evaluate rlhf models. motivated by the 52k instructions from alpaca @xcite , we leverage self-instruct @xcite to generate 106k additional instructions in english, introducing a larger dataset to facilitate rlhf evaluation. afterward, we utilize chat-gpt to translate the instructions into a diverse set of 26 languages, including high-, medium-, and low-resource languages (e.g., telugu, ukrainian, nepali, and kannada) to offer comprehensive re-sources and insights for multilingual instructiontuning. in addition, we introduce a translationbased prompt for chatgpt to produce rankings for multiple responses of the same instructions from the llms, which will be used to train the reward models for rlhf experiments. finally, we obtain the multilingual evaluation datasets for our fine-tuned llms by translating three benchmark datasets for llms in the widely-used huggingface open llm leaderboard @xcite into 26 languages, i.e., arc @xcite , hellaswag @xcite , and mmlu @xcite . using bloom @xcite and llama @xcite as the base llms, our experiments illustrate that rlhf generally performs better than sft for multilingual instruction tuning. we also highlight the greater challenges of lowresource languages for multilingual instructiontuning of llms that can be focused in future research. finally, we release our framework with the created resources and fine-tuned rlhf models. we also provide scripts to interact with our models at https://github.com/nlp-uoregon/okapi . ## data preparation a key requirement for our development of instruction-tuned llms with rlhf involves instruction, ranking, and evaluation datasets in multiple languages. to this end, we perform a comprehensive data collection process to prepare necessary data for our multilingual framework okapi in 26 languages, divided into four major steps: english instruction generation, instruction translation, ranking data production, and evaluation data creation. english instruction generation: an instruction example to tune llms often has three components: an instruction to specify the task, an input text, and an associated output text (i.e., demonstration or label) @xcite . as such, current public instruction datasets for llms mainly cover english or some popular languages. also, we note that a few recent instruction datasets such as xp3 @xcite and @xcite include multilingual data; however, their instructions are still written in english. additionally, these datasets tend to be converted from nlp datasets with template instructions, which cannot reflect the flexibility of human-written prompts @xcite . consequently, our goal is to develop instruction datasets with instructions, inputs, and output texts in multiple languages, including low-resource ones, to better realize general prompts from human. to achieve this goal, our strategy is to first obtain english instructions and then translate them into other languages. the benefits of our approach involve consistent instruction content across languages to facilitate performance comparison while taking advantages of translation systems to enable examination for more languages. as such, to conveniently scale our data, we follow the instruction generation method in alpaca, which in turn employs the self-instruct procedure in @xcite , to produce our english dataset. starting with a pool of 175 human-written seed instructions in english, at each time, alpaca samples several instructions from the seeds to form an in-context example to prompt the text-davinci-003 model of openai for new instruction generation. overall, alpaca releases 52k instructions for tuning llms. in this work, we apply the same self-instruct procedure as alpaca to generate 106k additional english instructions, resulting in a larger combined dataset of 158k instructions for our rlhf-based models in okapi. notably, we condition our generation process on the 52k instructions from alpaca so a new instruction is only saved if it is different enough from alpaca's and previous instructions per the rouge score criteria in alpaca @xcite . instruction translation: given the 158k english instructions, we aim to translate them into multiple other languages to obtain data for our multilingual models in okapi. we utilize chatgpt to translate the 158k english instructions into 26 target languages for okapi. compared to traditional machine translation systems, an advantage of chatgpt is the ability to use prompts to specify different expectations for the translated texts to facilitate diverse with their codes, numbers of first and second speakers (the "pop." column), data ratios in commoncrawl, and categories. the languages are grouped into categories based on their data ratios in commomcrawl: high-(h, > 1%), medium-(m, > 0.1%), and low-resource (l, > 0.01%). columns "b" and "l" indicate if a language is supported by the llms bloom and llama (respectively) or not. translation prompt: translate the values in the following json object @xmath3. you must keep the keys in the json object in english. if a value contains programming code, only translate the comments while preserving the code. your translations must convey all the content in the original text and cannot involve explanations or other unnecessary information. please ensure that the translated text is natural for native speakers with correct grammar and proper word choices. your translation must also use exact terminology to provide accurate information even for the experts in the related fields. your output must only contain a json object with translated text and cannot include explanations or other information. nlp tasks in english to avoid the exaggeration issues and achieve higher quality instructions. ## instruction-tuning with rlhf we follow three steps to develop a fine-tuned llm with rlhf for each target language in our okapi framework: supervised fine-tuning, reward model training, and reinforcement learning. supervised fine-tuning (sft): starting with a multilingual llm as the base, e.g., bloom @xcite , we fine-tune the model with our instruction dataset for the target language using supervised learning with the autoregressive objective. here, we fine-tune the entire base llm for all of its parameters with sft to accurately understand the model performance for multilingual settings. reward model training: the goal of this step is to train a reward model for the target language that will compute reward signals for reinforcement learning to further optimize the sft-tuned model from the previous step. for each pair of a prompt and potential response, our reward model returns a scalar value to quantify the appropriateness of the response with respect to the instruction and input text in the prompt. we exploit the instructions with multiple ranked responses in the data collection step for this training step. an example to train our reward model for a language involves an instruction and an input text (to form a prompt x) along with two sampled responses y c and y r for x from our datasets. based on the ranking information, we can assume one of the responses (i.e., y c ) is more preferable than the other (i.e., y r ). in the next step, the binary ranking loss @xcite is employed to train our reward model, aiming to assign a higher score r(x, y c ) for the preferred response y c than the score r(x, y r ) for y r : reinforcement learning (rl): with the reward model established for the target language, the sft model undergoes additional fine-tuning through rl to align it with human preferences. for this purpose, we employ the proximal policy optimization (ppo) algorithm @xcite that maximizes the mean reward of the model via the objective: here, d rl corresponds to the prompt distribution, and π ϕ (y|x) denotes the policy or language model that requires optimization. π ϕ (y|x) is initialized with the sft-tuned model π ϕ (y|x). also, kl(x, y) = d kl (π ϕ (y|x)||π 0 (y|x)) is the kullback-leibler divergence to penalize large deviation of π ϕ from the initial sft policy π 0 . ## experiments our okapi framework utilizes two multilingual llms: bloom @xcite and llama @xcite as the base models for the fine-tuning processes. we focus on their 7bparameter versions to facilitate the computing resources and achieve fairer comparison. for each base model and target language, we carry out both sft-based and rlhf-based instruction-tuning: following the huggingface open llm leaderboard, the eleuther ai language model evaluation harness framework @xcite is used to compute the model performance over the translated datasets arc, hellaswag, and mmlu for each language in our framework. as a reference, we also report the performance of the base models bloom and llama in the experiments. finally, for bloom, we further compare with bloomz @xcite , which is the fine-tuned version of bloom over the cross-lingual task mixture dataset xp3 with millions of multilingual instructions to achieve instruction-following ability. evaluation: tables 2 and 3 present the performance of the models on arc, hellaswag, and mmlu when bloom and llama are used as the base models (respectively). in the tables, for each language group (i.e., high-, medium-, and lowresource), we report the average performance over the languages and the performance for two example languages in the group. we also include the average performance over all languages in okapi. as some of our languages in okapi (especially the low-resource ones) are not supported by llama, the first observation from the tables is that rlhf is generally better than sft for multilingual fine-tuning of llms over different datasets, base models, and language groups. it is also evident that the rlhf-tuned models can significantly improve the performance of the original base models (i.e., bloom and llama) for almost all the language groups and datasets. in all, it highlights the quality of the generated instruction data and the effectiveness of rlhf in okapi. comparing the performance across language groups, the models tend to achieve the highest performance for the high-resource languages, followed by the medium-resource and low-resource languages. the performance improvement of rlhf for low-resource languages is also the least (based on bloom). interestingly, our fine-tuned bloom models with 158k generated instructions can significantly outperform bloomz over almost all the languages for the arc, hellaswag, and mmlu datasets using either sft or rlhf. as bloomz has fine-tuned bloom over more than 78m multilingual instructions converted from nlp datasets @xcite , it demonstrates the higher quality of our generated instructions for multilingual instruction tuning of llms. ## related work the most advanced methods for nlp involve finetuning the pre-trained language models (plms) on training data of the downstream tasks @xcite . instruction tuning can be considered as a special type of fine-tuning techniques for plms where generative plms (e.g., gpt) are further trained with instruction data to accomplish the instruction following abilities. sft is the most popular instruction tuning approach that is leveraged by most of the existing llms, including chatgpt, apaca @xcite , and vicuna @xcite . rlhf can also be used to further enhance llms @xcite although it has been less explored by current opensource llms due to the challenges in obtaining ranking data for the reward models. for multilingual learning, instruction tuning is only applied in the form of sft for non-english languages using multilingual llms, e.g., bloom and llama, in a few contemporary work @xcite @xcite . ## conclusion we present the first framework, called okapi, on instruction tuning for llms in multiple language using rlhf. we introduce instruction, ranked response, and evaluation data in 26 diverse languages to enable the training of rlhf methods. our results reveal the benefits of rlhf for multilingual fine-tuning of llms and the challenging problems of low-resource languages in this area.